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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Logistics) 
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•  Program  Executive  Officer,  SHIPS 
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•  Program  Executive  Officer,  Integrated  Warfare  Systems 
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Technology) 
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Abstract 

Initiatives  to  reduce  ship  maintenance  costs  have  not  yet  realized  the  normal  cost-reduction 
learning  curve  improvements.  One  explanation  is  the  lack  of  recommended  technologies. 
Damen,  a  Dutch  shipbuilding  and  service  firm,  has  incorporated  similar  technologies  and  is 
developing  others  to  improve  its  operations.  This  research  collected  data  on  Dutch  ship 
maintenance  operations  and  used  it  to  build  three  types  of  computer  simulation  models  of 
ship  maintenance  and  technology  adoption.  Results  were  compared  with  previously 
developed  modeling  results  of  U.S.  Navy  ship  maintenance  and  technology  adoption. 
Adopting  3D  PDF  alone  improves  ROI  significantly  more  than  adopting  a  logistics  package 
alone,  and  adding  both  technologies  improves  ROI  more  than  adding  either  technology 
alone.  Adoption  of  the  technologies  would  provide  cost  benefits  far  in  excess  of  not  using  the 
technologies,  and  there  were  marginal  benefits  in  sequentially  implementing  the  technologies 
over  immediately  implementing  them.  Potential  benefits  of  using  the  technologies  are  very 
high  in  both  cases.  Implications  for  acquisition  practice  include  the  need  for  careful  analysis 
and  selection  from  among  a  variety  of  available  information  technologies  and  the 
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recommendation  for  a  phased  development  and  implementation  approach  to  manage 

uncertainty. 

Introduction 

The  current  cost-constrained  environment  within  the  federal  government  and  the 
DoD  requires  a  defensible  approach  to  cost  reductions  without  compromising  the  capability 
of  core  defense  processes  and  platforms.  Due  to  this  environment,  defense  leaders  today 
must  maintain  and  modernize  the  U.S.  armed  forces  to  retain  technological  superiority  while 
simultaneously  balancing  defense  budget  cost  constraints  and  extensive  military  operational 
commitments.  At  the  same  time,  defense  leaders  must  navigate  a  complex  information 
technology  (IT)  acquisition  process.  Maintenance  programs  play  a  critical  role  in  meeting 
these  DoD  objectives.  One  such  core  process  that  is  central  to  U.S.  naval  operations  is  the 
ship  maintenance  process.  This  process  alone  accounts  for  billions  of  dollars  in  the  U.S. 
Navy’s  annual  budget.  There  have  been  a  series  of  initiatives  designed  to  reduce  the  cost  of 
this  core  process,  including  ship  maintenance.  SHIPMAIN,  and  its  derivatives,  was  one  of 
the  initiatives  designed  to  improve  ship  maintenance  performance  within  the  Navy  by 
standardizing  processes  in  order  to  take  advantage  of  learning  curve  cost  savings. 

However,  these  process  improvement  initiatives  have  not  yet  realized  the  normal 
cost-reduction  learning  curve  improvements  for  common  maintenance  items  for  a  series  of 
common  platform  ships.  One  explanation  is  that  the  initial  instantiation  of  SHIPMAIN  did  not 
include  two  recommended  technologies,  three-dimensional  laser  scanning  technology  (3D 
LST)  and  collaborative  product  life-cycle  management  (CPLM),  that  were  deemed 
necessary  by  the  creator  of  SHIPMAIN  for  ensuring  the  success  of  the  new  standardized 
approach  (i.e.,  normal  learning  curve  cost  savings).  Previous  research  (Ford,  Housel,  & 

Mun,  2011)  indicates  that  adding  these  technologies  may  help  SHIPMAIN,  or  its  derivatives, 
to  capture  the  potential  savings.  But  the  technologies  have  not  been  implemented  to  date  in 
the  ship  maintenance  processes. 

However,  Damen,  a  large  shipbuilding  and  service  firm  has  incorporated  similar 
technologies  and  is  developing  others  to  improve  its  operations.  In  addition,  the  Royal  Dutch 
Navy  (RDN)  performs  all  of  its  own  ship  maintenance  in  a  single  yard  and  operation.  In  the 
current  study,  the  potential  benefits  of  similar  technologies  are  extrapolated  and  compared 
with  similar  projections  for  U.S.  Navy  ship  maintenance  processes.  These  organizations 
provide  a  source  of  relatively  reliable  data  on  operations  that  are  comparable  to  those 
performed  by  the  U.S.  Navy. 

Problem  Description 

Previous  research  on  the  potential  use  of  3D  LST  and  CPLM  technology  in  U.S. 

Navy  ship  maintenance  (e.g.,  Komoroski,  2005;  Ford,  Housel,  &  Mun,  2011)  estimated  the 
impacts  on  processes  due  to  technology  adoption.  Changes  such  as  reengineering  ship 
maintenance  processes,  the  sizes  of  reductions  in  cycle  times,  and  workforce  requirements 
are  examples  of  model  portions  that  required  modelers  to  make  assumptions  about  the 
potential  impacts  of  these  technologies  in  modeling  projected  results.  While  the  previous 
work  has  provided  defensible  estimates  of  potential  improvements  (in  returns  on  investment, 
ROI)  and  cost  savings,  the  validity  and  usefulness  of  these  models  has  been  limited  by  the 
lack  of  comparative  data  on  ship  maintenance  processes  and  technology  investments,  and 
of  their  potential  impacts  on  performance.  Therefore,  the  acquisition  of  data  on  Dutch  naval 
fleet  maintenance  processes  and  the  comparison  of  those  data  with  previous  U.S.  Navy 
results  were  critical  steps  in  improving  U.S.  naval  technology  acquisition  decision-making,  in 
particular  with  regard  to  ship  maintenance. 
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To  be  valuable,  the  data  source  or  sources  for  this  work  had  to  have  several  critical 
similarities  with  U.S.  naval  ship  maintenance  processes.  The  data  source  had  to  consider 
technological  innovation  and  the  adoption  of  advanced  technologies  to  be  an  important  part 
of  its  naval  maintenance  acquisition  strategy.  The  data  source  or  sources  had  to  be  large 
enough  to  support  continuous  ship  maintenance  operations  because  the  intermittent 
stopping  and  restarting  of  operations  would  not  be  consistent  with  important  assumptions  of 
the  modeling  approach.  Finally,  the  data  source  had  to  be  accessible,  willing  to  share  the 
data,  and  willing  to  allow  us  to  obtain  the  new  data  required  for  our  modeling  approach. 
These  and  other  criteria  limited  the  potential  pool  of  sources  to  nations  or  large  industrial 
ship  maintenance  organizations  that  were  on  good  terms  with  the  United  States,  advanced 
enough  in  their  operations  to  compare  with  those  of  the  U.S.  Navy,  progressive  enough  in 
their  strategies  to  include  continuous  technology  adoption,  and  willing  to  share  data  and 
information  that  is  often  considered  essential  for  national  security  or  competitive  advantage. 
Damen  Industries  and  the  RDN  met  most  of  these  criteria  and  were  willing  to  meet  our 
requirements  for  data  acquisition  and  sharing. 

The  current  work  addresses  the  following  questions: 

•  How  are  the  Dutch  using  and  preparing  to  adopt  advanced  technologies, 
such  as  3D  LST  and  CPLM,  in  shipbuilding  and  maintenance? 

•  What  are  the  potential  changes  in  ROIs  provided  by  the  adoption  of  these 
advanced  technologies? 

•  How  do  those  potential  returns  compare  with  projected  estimates  of  returns 
on  technology  adoption  of  3D  LST  and  CPLM  in  the  U.S.  Navy? 

Research  Methodology  and  Background1 

The  traditional  ROI  equation  is  typically  expressed  as  (revenue  - 
investmentj/investment,  which  represents  the  productivity  ratio  of  output  (i.e.,  revenue  in 
ROI  -*■  input  or  investment  cost  in  ROI).  Accomplishing  this  analysis  in  a  nonprofit 
environment  presents  challenges  because  there  is  no  actual  revenue  generated.  Cost 
savings  from  reductions  in  manpower  requirements  (i.e.,  time  allocated  to  employee 
workload  for  various  tasks)  is  available  to  provide  the  impact  on  the  denominator  of  the  ship 
maintenance  efforts.  The  knowledge  value  added  (KVA)  methodology  (Housel  &  Kanevsky, 
1995)  also  allows  for  generation  of  a  quantifiable  surrogate  for  revenue  in  the  form  of 
common  units  of  output  described  in  terms  of  units  of  learning  time.  Specifically,  the  KVA 
methodology  allowed  the  study  team  to  quantify  the  knowledge  embedded  in  the  new 
processes  to  use  in  generating  common  units  of  output  estimates.  The  KVA  analysis 
provided  the  basic  ROI  estimates  critical  in  forecasting  the  future  value  of  various 
automation  options. 

The  system  dynamics  methodology  was  used  to  model  the  impacts  of  automation  on 
operations.  System  dynamics  applies  a  control  theory  perspective  to  the  design  and 
management  of  complex  human  systems.  System  dynamics  combines  servo-mechanism 
thinking  with  computer  simulation  to  create  insights  about  the  development  and  operation  of 
these  systems.  Forrester  (1961)  developed  the  methodology’s  philosophy,  and  Sterman 
(2000)  specified  the  modeling  process  with  examples  and  described  numerous  applications. 
System  dynamics  is  used  to  build  causal-based  (versus  correlation-based)  models  that 
reflect  the  components  and  interactions  that  drive  behavior  and  performance.  The 
methodology  has  been  used  extensively  to  explain,  design,  manage,  and,  thereby,  improve 


1  See  Ford,  Housel,  and  Mun  (2012)  for  a  more  detailed  description  of  the  research  methodologies 
applied. 
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the  performance  of  many  types  of  systems,  including  development  projects.  The 
methodology’s  ability  to  model  many  diverse  system  components  (e.g.,  work,  people, 
money,  value),  processes  (e.g.,  design,  technology  development,  production,  operations, 
quality  assurance),  and  managerial  decision-making  and  actions  (e.g.,  forecasting,  resource 
allocation)  makes  system  dynamics  useful  for  modeling  and  investigating  military  operations, 
the  design  of  materiel,  and  acquisition. 

The  integrated  risk  management  (IRM)  framework  and  supporting  toolset  was  used 
to  optimize  the  portfolio  over  time.  IRM  is  an  eight-step,  quantitative,  software-based 
modeling  approach  for  the  objective  quantification  of  risk  (cost,  schedule,  technical), 
flexibility,  strategy,  and  decision  analysis.  The  method  can  be  applied  to  program 
management,  resource  portfolio  allocation,  return  on  investment  to  the  military  (maximizing 
expected  military  value  and  objective  value  quantification  of  nonrevenue  government 
projects),  analysis  of  alternatives  or  strategic  flexibility  options,  capability  analysis,  prediction 
modeling,  and  general  decision  analytics.  The  method  and  toolset  provide  the  ability  to 
consider  hundreds  of  alternatives  with  budget  and  schedule  uncertainty  and  provide  ways  to 
help  the  decision-maker  maximize  capability  and  readiness  at  the  lowest  cost.  This 
methodology  is  particularly  amenable  to  resource  reallocation  and  has  been  taught  and 
applied  by  the  authors  for  the  past  10  years  at  over  100  multinational  corporations  and  over 
30  projects  at  the  U.S.  Department  of  Defense  (DoD). 

The  research  team  collected  data  on  Dutch  ship  operations  as  described  in  the 
section  titled  Data  Collection  Methods  and  used  it  to  build  three  types  of  computer 
simulation  models  of  ship  maintenance  and  technology  adoption:  KVA  models  of  return  on 
technology  investments  in  those  operations,  system  dynamics  models  (SD)  of  ship 
maintenance  operations,  and  IRM  models  of  implementation  plans  for  technology  adoption. 
The  results  were  then  analyzed  and  compared  with  previously  developed  modeling  results 
of  U.S.  Navy  ship  maintenance  and  technology  adoption. 

Data  Collection  Methods 

Data  on  the  practices  of  Dutch  industry  and  naval  ship  maintenance  proved  very 
difficult  and  time  consuming  to  obtain.  Initial  contact  with  Dutch  industry  participants  and 
ship  maintenance  technology  providers  developed  slowly  over  several  months  into 
relationships  that  eventually  led  to  data  collection  opportunities.  Several  sources  of  data 
were  utilized,  including  a  Dutch  shipbuilder  (Damen)  and  the  RDN.  Data  on  the  use  of 
technology  in  Dutch  fleet  maintenance  was  collected  by  two  primary  methods:  (1)  in-person 
interviews  and  meetings  with  managers  of  the  leading  corporation  in  the  Dutch  shipbuilding 
industry  (Damen)  and  with  officers  and  civilian  employees  of  the  RDN,  and  (2)  tours  of  three 
Dutch  shipbuilding  and  maintenance  facilities. 

In-person  interviews  and  meetings  with  managers  at  Damen  and  with  officers  and  a 
civilian  employee  of  the  RDN  occurred  during  a  data  collection  trip  by  one  of  the  research 
team  members  (Ford)  to  the  Netherlands  in  June  2012,  as  did  the  tours  of  Dutch  ship 
building  and  maintenance  facilities.  Meetings,  semi-structured  interviews,  and  extended 
discussions  were  held  with  six  managers  of  Damen  Industries  and  the  RDN  in  three 
locations  over  three  days.  At  these  meetings,  Damen  managers  made  presentations  on 
Damen’s  operations,  uses  of  technologies,  investigations  of  specific  technologies  for 
potential  development  and  adoption  (including  3D  LST  and  CPLM  software),  Integrated 
Logistics  System  (ILS),  and  information  technology  products  under  development  for  use  in 
ship  maintenance.2  Separately,  a  meeting  and  semi-structured  interview  was  conducted  with 


2  Copies  of  these  presentations  were  requested,  but  not  provided.  Data  collection  results  are  based 
on  notes  taken  by  the  investigator  during  the  meetings,  interviews,  and  tours  of  facilities. 
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the  two  RDN  officers  responsible  for  ship  maintenance  at  the  RDN  shipyard  at  Nieuwe 
Haven  in  Den  Helder.  Tours  of  the  RDN  fleet  maintenance  facility  in  Nieuwe  Haven  and  two 
Damen  shipyards  were  provided  during  the  data  collection  trip. 

Data  Collection  Results 

Damen’ s  Use  of  Technology 

The  Damen  Shipyards  Group  (www.damen.nl/)  is  a  large  Dutch  shipbuilding  firm  with 
worldwide  operations  (11  shipyards  with  five  outside  The  Netherlands).  The  firm  was  started 
in  1922  by  Jan  and  Rien  Damen.  The  firm  grew  substantially  after  Kommer  Damen  (the 
current  owner)  bought  it  in  1969  and  introduced  modular  and  standardized  shipbuilding  to 
the  industry.  The  firm  now  employs  over  6,000  people  and  builds  an  average  of  150  vessels 
per  year.  The  firm  obtained  Damen  Schelde  in  2000,  which  focuses  exclusively  on  naval 
ship  design,  building,  and  maintenance.  Damen  Schelde  manufactures  an  average  of  one  to 
two  ships  per  year,  employs  about  550  people,  and  performs  about  €210  million  per  year. 
Damen  Schelde  acts  as  the  prime  contractor  and  integrator  on  its  shipbuilding  projects, 
utilizing  many  subcontractors.  Although  Damen  Schelde  provides  ship  maintenance 
services  to  its  international  (i.e.,  not  Dutch)  customers,  it  does  not  provide  any  ship 
maintenance  services  for  the  RDN. 

Damen  Schelde  has  used  an  ILS  since  2002  to  manage  the  shipbuilding  process 
from  project  initiation  through  the  development  of  a  logistics  plan  for  customers.  The  ILS  is 
the  plan  for  the  development  of  a  ship  and  includes  ship  design;  production;  quality 
assurance,  quality  control  (QAQC);  training  of  ship  operators;  and  coordination  with 
customers.  The  ILS  does  not  include  service  contracts  or  life-cycle  costs  due  to  the  difficulty 
of  forecasting  those  costs.  The  focus  of  the  ILS  is  to  provide  maximum  ship  operational 
availability,  reliability,  and  maintainability.  It  does  this  partially  by  using  a  single  point  of 
contact  within  Damen  throughout  the  project  who  manages  an  interdisciplinary  team  (e.g., 
engineering,  work  preparation,  procurement,  service).  Damen  Schelde  currently  uses  a 
variety  of  information  technologies  to  facilitate  their  ILS  approach  to  shipbuilding  and  is 
constantly  investigating  new  technologies  that  may  improve  its  design  and  manufacturing. 

Of  particular  relevance  to  the  current  work,  Damen  Schelde  uses  four  separate  software 
products  to  manage  its  shipbuilding:  an  advanced  three-dimensional  CADD  program  for 
design,  a  CPLM  product  as  a  database  for  ship  components,  an  Enterprise  Resource 
Planning  (ERP)  system,  and  a  software  tool  for  scheduling.  The  latter  three  of  these 
systems  are  connected  to  users  with  a  project  information  portal  developed  by  Damen 
Schelde.  The  informant  reported  that  Damen  developed  the  portal  because  the  CPLM 
product  did  not  include  adequate  user  interfaces. 

Damen  Schelde  has  investigated  and  is  currently  investigating  other  technologies  for 
potential  adoption.  Four  technologies  were  described  and  discussed: 

1 .  3D  LST :  This  technology  was  investigated  but  was  assessed  to  currently  be 
too  immature  for  adoption  by  Damen  Schelde.  The  investigation  included  a 
discussion  of  the  current  use  of  the  technology  in  the  automobile  industry,  as 
well  as  its  potential  use  to  scan  engine  rooms  and  for  floor  flattening.  The  use 
of  360-degree  photography  (often  used  in  conjunction  with  3D  LST)  was 
considered  by  Damen  Schelde  as  a  potential  tool  for  training  (see  Komoroski, 
2005,  for  more  details  on  3D  LST). 

2.  3D  PDF  files:  Three-dimensional  animated  “movies”  of  shipbuilding  can  be 
created  in  a  PDF  format  (by  Adobe  Acrobat®)  and  sent  to  shipyards  for  use 
in  the  field  by  craftsmen  who  view  the  file  on  an  electronic  reader  (e.g.,  an 
iPad®).  The  files  would  replace  flat  drawings  for  use  in  construction.  The  file 
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visually  communicates  the  sequence  of  building  (or  maintenance)  operations 
and  components,  and  operations  can  have  notes  attached  to  them  that 
provide  additional  information  (e.g.,  part  numbers  or  warnings  of  special 
issues).  The  ability  to  animate  these  files  allows  engineers  to  visually  show 
craftsmen  sequences  of  operations,  routes  of  access  and  egress  for  line 
replaceable  units  (LRU3),  and  other  information  that  is  difficult  or  impossible 
to  show  with  traditional,  static,  two-dimensional  drawings.  The  use  of  this 
technology  shifts  the  understanding  of  the  design  intention  from  the  designers 
(in  the  Netherlands)  to  the  shipbuilding  yard  (typically  in  other  countries 
around  the  world).  The  use  of  visual  information  (the  animation  of  steps)  is 
expected  to  greatly  improve  communication  across  languages  since  many  of 
the  craftsmen  in  Damen’s  shipyards  do  not  read  English  well.  Damen 
considers  improvements  in  information  content  communicated  to  be  the 
primary  benefit  of  this  system  (versus  cost  savings).  Damen  Schelde  is  very 
optimistic  about  the  potential  for  this  technology  to  improve  its  operations  and 
is  actively  working  on  developing  it  (e.g.,  selecting  software,  addressing  the 
importing  of  the  3D  design  drawings).  Generating  the  animated  files  and 
adding  the  building  steps  to  the  design  files  is  expected  to  be  relatively  easy 
once  the  system  has  been  developed. 

3.  SIGMA  Shipbuilding  Strategy:  This  is  a  standardized  process  for  creating  a 
ship  that  spans  from  design  through  materials  procurement,  production,  and 
testing  of  a  ship.  The  key  feature  of  the  strategy  is  the  use  of  modular  ship 
sizes  and  systems  that  can  be  easily  adapted  to  specific  customer  needs.  For 
example,  Damen  Schelde  has  disaggregated  an  entire  ship  into  five 
standardized  modules  (e.g.,  fore,  midship,  aft)  with  major  systems  located  in 
specific  sections.  Each  module  is  considered  a  subproject.  As  an  example  of 
an  advantage  provided  by  the  strategy,  the  modules  and  their  interfaces  are 
designed  such  that  the  ship  can  be  made  longer  by  adding  an  additional 
midsection.4 

4.  Radio  Frequency  Identification  (RFID):  This  established  technology  is  being 
considered  for  use  to  improve  Damen’s  supply  chain  management.  Primary 
benefits  are  believed  to  be  improved  value  of  information  and  a  reduction  in 
the  duration  for  getting  information  into  Damen  databases  (e.g.,  warehouse 
contents,  components  on  specific  ships).  Both  passive  and  active  tags  are 
being  considered. 

Damen  Services  also  develops  advanced  technologies  for  use  by  Damen 
Enterprises.  Damen  Services  focuses  on  providing  ongoing  maintenance  parts  and  services 
to  Damen  customers  after  a  ship  has  been  designed,  built,  and  delivered,  but  also  provides 
other  services  such  as  civil  works  (e.g.,  wharves  and  storage  facilities). 

The  Maintenance  and  Spares  department  maintains  information  on  ship 
configuration  (using  an  ERP  system),  parts  inventories,  spare  parts  packages,  and 
maintenance  management  systems.  It  also  provides  information  technology  support  for 
Damen.  It  is  developing  a  web  portal  for  clients  that  will  allow  clients  access  to  Damen-held 
data  on  each  of  the  customer’s  ships  down  to  the  individual  component  level.  This  will 


3  Line  replaceable  unit  is  a  commonly  used  term  in  manufactured  devices  for  any  modular  component 
that  is  designed  to  be  interchangeable. 

4  This  portion  of  the  SIGMA  strategy  applies  the  Boeing  strategy  for  the  design  and  production  of  the 
737  that  has  different  lengths  to  shipbuilding. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-9- 


partially  be  accomplished  with  a  work  breakdown  system  (WBS)  that  disaggregates  a  ship 
or  system  into  product  parts  (e.g.,  engine,  bilge  pump)  and  a  functional  breakdown  system 
that  disaggregates  the  ship  into  functions  (e.g.,  port  propulsion)  that  are  met  with  a  product 
part  (in  the  WBS)  and  have  an  associated  maintenance  schedule,  which  includes  monitoring 
measurements  and  frequency,  parts  documentation,  and  so  forth.  The  WBS  has  three 
levels:  subsystems  (e.g.,  propulsion,  hoisting),  with  a  typical  ship  having  20-70  subsystems; 
Level  2  parts  (e.g.,  pump,  shaft),  with  about  1,000  per  ship;  and  Level  3  parts  (e.g.,  bolt, 
flange),  with  70,000-80,000  per  ship. 

This  system  will  be  linked  with  an  online  parts  ordering  portal  so  that  customers  can 
order  parts  from  Damen  (similar  to  Amazon’s  online  selling  of  books,  etc.).  Damen  Services 
plans  to  use  the  information  (e.g.,  the  frequency  orders  for  specific  components)  captured 
through  this  system  to  develop  maintenance  optimization  information.  Damen  Services 
envisions  three  types  of  maintenance:  corrective  maintenance  (after  the  component  needs 
work),  preventative  maintenance  (based  on  forecasts  of  maintenance  needs),  and  condition- 
based  maintenance  (based  on  actual  conditions  of  components).  Condition-based 
maintenance  is  an  optimized  version  of  preventative-based  maintenance  that  is  currently 
under  development.  It  requires  sensors  to  collect  data  on  component  conditions  that  will  be 
used  to  generate  condition  assessments. 

Royal  Dutch  Navy  Fleet  Maintenance 

Data  collection  directly  from  the  RDN  was  particularly  valuable  for  at  least  two 
reasons.  First,  as  the  navy  of  a  sovereign  country  with  objectives  that  are  similar  to  those  of 
the  United  States,  the  objectives  and  issues  of  the  RDN  are  more  likely  to  match  those  of 
the  U.S.  Navy  than  those  of  some  other  nations.  Data  collection  supported  this  assumption. 
For  example,  technology  leadership,  interoperability,  and  reliability  in  meeting  operational 
needs  are  paramount  to  the  RDN,  and  the  RDN  has  recently  experienced,  and  expects  to 
continue  to  experience,  reductions  in  budgets  just  as  is  the  case  with  the  U.S.  Navy.  The 
Dutch  navy  continues  to  face  budget  cuts  and  increasing  technology  needs,  is  currently  in 
reorganization  to  reduce  total  workforce  (internal  to  the  navy  and  civilian  naval  workforce)  by 
20%,  and  is  transferring  from  legacy  information  systems  to  an  integrated  ERP  system  for 
maintenance  operations.  Also,  the  RDN  performs  all  of  the  maintenance  on  its  fleet,  thereby 
making  it  the  primary  data  source  concerning  RDN  fleet  maintenance  process  performance. 

The  interviews  with  the  two  RDN  officers  in  the  Naval  Maintenance  and  Service 
Agency  provided  a  general  introduction  to  the  issues  faced  by  the  Dutch  navy  in  building 
and  maintaining  its  fleet.  The  RDN  addresses  its  challenges  by  means  similar  to  those  used 
by  the  U.S.  Navy,  such  as  waiting  for  technology  to  mature  (technology  readiness  level 
[TRL]  >  7  before  adoption)  and  incremental  capability  increases  based  on  budgets. 
Noticeably  different,  both  the  RDN  and  Damen  described  the  critical  role,  and  standard 
Dutch  practice,  of  adjusting  requirements  to  meet  budgets  in  shipbuilding.  The  RDN  is 
facing  increasing  pressure  to  control  life-cycle  costs  in  its  fleet,  which  are  largely  driven  by 
personnel  and  fuel.  This  has  led  it  to  approve  significantly  stricter  operations  manning 
requirements  for  ship  design  (i.e.,  lower  maximum  shipboard  personnel),  which  has  driven 
Damen  to  increase  the  use  of  automation  in  its  ship  designs. 

The  primary  informant  on  RDN  fleet  maintenance  operations  provided  a  diagram  of 
those  operations  (see  Figure  1)  and  a  written  description  of  each  of  the  steps  identified  in 
the  diagram. 
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Logistics  Process  Royal  Netherlands  Navy 


Figure  1.  Diagram  of  Royal  Dutch  Navy  Fleet  Maintenance  Processes 

(P.  Kense,  personal  communication,  June  21, 2012) 

The  process  steps  shown  in  Figure  1  were  described  in  writing  by  the  informant  with 
the  following  list.5  In  the  list,  the  abbreviation  LRU  stands  for  line  replaceable  unit,  a 
commonly  used  term  in  the  area  of  manufactured  devices  for  any  modular  component  that  is 
designed  to  be  interchangeable. 

Logistic  Process  Royal  Netherlands  Navy 

1.  In  case  LRU  fails  the  on-board  personnel  will  replace  this  LRU  by  a  spare 
(on-board;  OLM  qualification  required). 

2.  The  defect  LRU  will  be  send  to  the  warehouse,  and  a  “new”  LRU  will  be  send 
to  the  ship. 

3.  The  defect  LRU  will  be  send  to  the  Naval  Maintenance  Establishment  (NME) 
for  repair.  After  the  LRU  is  repaired  it  will  be  send  to  the  warehouse  again  “as 
good  as  new”  (DLM  qualification  required). 

4.  If  the  NME  needs  parts  to  repair  an  LRU,  the  parts  will  be  extracted  from  the 
industry,  when  the  NME  is  not  able  to  repair  this  LRU,  it  can  be  send  to  the 
manufacturer.  Also,  manpower  can  be  hired  to  fix  problems. 

5.  If  spare  is  not  available,  sometimes  it  will  be  cannibalized  from  another  ship. 

6.  If  the  on-board  personnel  is  not  able  to  fix  the  problem  by  themselves  (due  to 
the  complexity  of  the  failure)  assistance  from  the  NME  is  needed  (ILM 
qualification  required). 

7.  If  the  problem  is  too  complex  for  the  NME  also,  the  industry  can  be  hired  to 
solve  this  problem. 

The  following  seven  process  steps  were  elaborated  on  by  the  informant  (the 
abbreviations  DLM,  OLM,  and  ILM  refer  to  Dutch  terms  for  training  levels): 


5  The  process  step  descriptions  have  been  transcribed  exactly  as  provided  in  English  by  the  RDN, 
including  uncommon  English  grammar  and  spelling. 
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Step  1:  Performed  onboard,  for  example  to  provide  operational 
maintenance  of  weapons  systems 

Step  2:  Purely  a  transit  operation  that  requires  only  a  truck  driver  (if  ship  is 
in  port) 

Step  4:  Requires  DLM  level  of  training 

Step  5:  Requires  OLM  level  of  training 

Step  6:  Requires  ILM  level  of  training  (=  LTS  +  MTS  +  10-25  days  of 
training) 

Step  7:  Requires  DLM  level  of  training 

Fleet  maintenance  for  the  RDN  requires,  at  a  minimum,  completion  of  education  at  a 
Lower  Technical  School  (LTS)  and  a  Middle/Intermediate  Technical  School  (MTS).  The  LTS 
is  typically  attended  between  ages  12-16,  and  the  MTS  is  typically  attended  between  ages 
1 6-21 .  After  completion  of  LTS  and  MTS,  future  RDN  ship  fleet  maintenance  personnel 
must  complete  at  least  one  of  three  other  forms  of  training. 

System  Dynamics  Model  Structure 

The  system  dynamics  model  simulates  the  movement  of  LRU  among  the  various 
locations  where  they  are  used,  stored,  or  repaired.  Each  flow  of  LRUs  between  two  stocks 
represents  the  processing  rate  of  one  of  the  process  steps  in  a  KVA  model.  A  simplified 
diagram  of  the  stocks  and  flows  of  the  model  are  shown  in  Figure  2.  Boxes  represent 
stocks,  or  accumulations  of  LRU.  Each  stock  in  Figure  2  represents  a  location  in  Figure  1, 
plus  on-board  LRU  storage  as  a  separate  LRU  accumulation.  Arrows  with  valve  symbols  in 
Figure  2  represent  the  movement  of  LRUs  between  stocks.  Numbers  in  parentheses  in  the 
titles  of  flows  represent  the  process  steps  shown  in  Figure  1  (ovals  with  arrows)  and  the 
KVA  model  process  steps  (described  later). 


j&:e(4) 
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Figure  2.  Royal  Dutch  Navy  Ship  Maintenance:  Stocks  and  Flows  of  the  System 

Dynamics  Model 

The  sizes  of  the  flows  in  the  system  dynamics  model  describe  the  rate  of  movement 
of  LRUs  among  the  stocks.  Therefore,  the  simulated  flows  in  the  system  dynamics  model 
become  direct  inputs  to  the  “times  processed  per  year”  portion  of  the  KVA  models.  Flow 
rates  were  modeled  to  reflect  the  sequence  of  processes  in  operations.  For  example,  in 
normal  operations,  the  replacement  of  a  broken  LRU  in  an  operating  ship  with  one  from  the 
ship’s  on-board  storage  (“Replace  broken  LRU  from  storage  [1]”  on  the  left  of  Figure  2) 
would  be  followed  by  the  broken  LRU  in  storage  being  replaced  by  an  operational  LRU  from 
the  warehouse  (“Replace  broken  LRU  from  warehouse  on  onboard  storage  [8]”  at  the  top  in 
Figure  2).  This  replacement  would  be  followed  by  the  broken  LRU  being  sent  to  the  NME 
where  it  would  be  repaired  and  returned  to  the  warehouse  (“NME  repairs  broken  LRU  in 
warehouse  [3]”  on  the  right  in  Figure  2).  These  precedencies  are  modeled  by  having  the 
downstream  process  equal  to  its  preceding  process  step  with  a  delay  that  reflects  the  transit 
and  subsequent  processing  time.  Some  flows  (e.g.,  “NME  repairs  broken  LRU  from 
warehouse  [3]”)  are  aggregations  of  multiple  upstream  flows.  Core  flows  are  based  on  the 
mean  time  between  failure  of  LRUs  and  the  fraction  of  failures  addressed  with  each 
process. 

The  system  dynamics  model  was  calibrated  to  reflect  RDN  ship  maintenance  (see 
Ford,  Housel,  &  Mun,  2012,  for  details). 

Knowledge  Value  Added  Models  to  the  Royal  Dutch  Navy  Ship  Maintenance 

Four  KVA  models  were  built  based  on  the  RDN  ship  maintenance  processes  (see 
Ford,  Housel,  &  Dillard,  2010,  for  details  and  examples  of  KVA  modeling): 

1.  Baseline  RDN  ship  maintenance  processes 

2.  Baseline  RDN  ship  maintenance  processes  changed  to  reflect  the  adoption 
and  use  of  a  logistics  package  from  an  integrated  CPLM  system  such  as  was 
investigated  by  Damen 

3.  Baseline  RDN  ship  maintenance  processes  changed  to  reflect  the  adoption 
and  use  of  3D  PDF  modeling  managed  with  a  CPLM  system  as  planned  by 
Damen 

4.  Baseline  RDN  ship  maintenance  processes  changed  to  reflect  the  adoption 
and  use  of  a  logistics  package  and  3D  PDF  modeling  managed  by  an 
integrated  CPLM  system 

Model  Simulations  and  Results 

The  system  dynamics  model  was  simulated  to  represent  the  four  technology 
adoption  scenarios  described  in  the  previous  section.  The  output  of  each  system  dynamics 
model  simulation  was  used  as  input  to  a  KVA  model.  Those  KVA  models  were  then  used  to 
estimate  the  ROI  of  each  process  in  each  of  the  four  scenarios  and  the  cumulative  ROI  for 
each  scenario.  The  results  based  on  the  models  and  their  calibrations  are  shown  in  Table  1. 
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Table  1.  Knowledge  Value  Added  Model  Results 


Return  On  Investment  (ROI) 

Process 

Description 

Baseline 

Add 

Logistics 

Add  3D 

PDF 

Add 

Logistics 
&  3D  PDF 

1 

Replace  LRU  with  on-board 
spare 

90% 

261% 

501% 

464% 

2 

Replace  operating  LRU  with 
warehouse  spare 

90% 

151% 

621% 

1 027% 

3 

NME  repairs  warehouse  LRU 
and  returns  it  to  warehouse 

8% 

65% 

95% 

236% 

4 

Manufacturer  repairs  LRU  for 
NME  &  it  returns  to  warehouse 

31% 

88% 

168% 

1 68% 

5 

Replace  on-board  LRU  with 

LRU  cannibalized  from  another 
ship 

90% 

151% 

621% 

1 027% 

6 

NME  repairs  on-board  LRU  and 
returns  it  to  ship 

265% 

10% 

99% 

1 92% 

7 

Industry  repairs  on-board  LRU 
and  returns  it  to  ship 

34% 

1 78% 

1 35% 

318% 

8 

Replace  on-board  storage  LRU 
with  warehouse  spare  (transit 
only) 

301% 

759% 

759% 

759% 

9 

Replace  cannabalized  LRU  with 
warehouse  spare  (transit  only) 

140% 

329% 

862% 

1 1 02% 

TOTAL  ALL  PROCESSES 

35% 

77% 

1 35% 

274% 

Although  increased  throughput  due  to  reduced  processing  durations  (which  increase 
the  ROI  numerator)  can  partially  explain  differences  in  the  ROI  in  Table  1,  cost  reduction 
(which  decreases  the  ROI  denominator)  is  the  primary  driver  of  increases  in  ROI.  For 
example,  Processes  8  and  9  are  benefited  by  reductions  in  rework  (e.g.,  errors  in 
transporting  LRU)  due  to  the  adoption  of  a  logistics  package.  This  reduces  the  number  of 
transport  trips  required  (the  function  of  these  processes),  thereby  significantly  reducing 
costs  and  increasing  the  ROI.  In  contrast,  Processes  3,  4,  and  6  are  highly  skilled  processes 
that  are  difficult  to  replace  with  technology  and,  therefore,  benefit  less  from  technology 
adoption  than  other  processes.  This  results  in  a  smaller  increase  in  ROI  for  these 
processes. 

Analysis  of  Simulation  Model  Results 

A  variance  analysis  was  performed  on  the  KVA  model  results  (Table  1)  to  evaluate 
the  relative  impacts  of  the  adoption  of  different  technologies  (Table  2).  ROIs  for  each  of  the 
three  technology  adoption  alternatives  were  compared  with  the  baseline  ROIs  to  estimate 
improvement  due  to  technologies  (see  the  left  three  columns  of  results  in  Table  2).  In 
addition,  the  improvement  from  adopting  both  technologies  over  adopting  only  the  3D  PDF 
technology  was  estimated  (see  the  right  column  in  Table  2). 
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Table  2. 


Variance  Analysis  of  Knowledge  Value  Added  Model  Results 


Return  On  Investment  (ROI) 

Process 

Description 

Add  Logistics 
Improvement 
over  Baseline 

Add  3Dpdf - 
Improvement 
over  Baseline 

Add  Logistics 
&  3Dpdf  - 
Improvement 
over  Baseline 

Add  Logistics  & 
3Dpdf  - 
Improvement 
over  adding  only 
3Dpdf 

1 

Replace  LRU  with  on-board 
spare 

171% 

411% 

374% 

-38% 

2 

Replace  operating  LRU  with 
warehouse  spare 

61% 

532% 

937% 

406% 

3 

NME  repairs  warehouse  LRU 
and  returns  it  to  warehouse 

57% 

87% 

227% 

140% 

4 

Manufacturer  repairs  LRU  for 
NME  &  it  returns  to  warehouse 

57% 

138% 

138% 

0% 

5 

Replace  on-board  LRU  with 

LRU  cannibalized  from  another 
ship 

61% 

532% 

937% 

406% 

6 

NME  repairs  on-board  LRU  and 
returns  it  to  ship 

-256% 

-166% 

-73% 

93% 

7 

Industry  repairs  on-board  LRU 
and  returns  it  to  ship 

145% 

101% 

284% 

183% 

8 

Replace  on-board  storage  LRU 
with  warehouse  spare  (transit 
only) 

458% 

458% 

458% 

0% 

9 

Replace  cannabalized  LRU  with 
warehouse  spare  (transit  only) 

189% 

721% 

962% 

240% 

TOTAL  ALL  PROCESSES 

42% 

100% 

239% 

139% 

Referring  to  Table  2,  adding  either  or  both  of  the  technologies  improves  overall  ship 
maintenance  ROI,  as  indicated  by  the  positive  numbers  in  the  last  row  of  Table  2.  Adopting 
3D  PDF  alone  improves  ROI  significantly  more  than  adopting  a  logistics  package  alone 
(100%  improvement  >  46%  improvement),  and  adding  both  technologies  improves  ROI 
more  than  adding  either  technology  alone  (239%  improvement  >  42%  improvement  or  100% 
improvement),  suggesting  that  there  may  be  synergy  between  the  technologies.  This  is  also 
supported  by  the  139%  improvement  by  adding  logistics  if  3D  PDF  is  already  in  place  (see 
the  lower  right  result  in  Table  2). 

Adopting  the  technologies  does  not  impact  the  ROI  of  individual  processes  equally. 
Among  the  seven  core  processes  (1-7),  adding  only  a  logistics  package  (see  the  left  column 
of  results  in  Table  2)  increases  the  “Replace  LRU  with  on-board  spare”  (Process  1)  most,  by 
171%,  and  decreases  the  return  of  Process  6,  “NME  repairs  on-board  LRU  and  returns  it  to 
ship,”  by  256%.  Among  the  seven  core  processes,  adding  only  3D  PDF  increases 
Processes  2  and  5,  “Replace  operating  LRU  with  warehouse  spare”  and  “Replace  on-board 
LRU  with  LRU  cannibalized  from  another  shop”  most,  by  532%,  and  decreases  the  return  of 
Process  6,  “NME  repairs  on-board  LRU  and  returns  it  to  ship”  by  166%.  Among  the  seven 
core  processes,  adding  both  technologies  increases  Processes  2  and  5,  “Replace  operating 
LRU  with  warehouse  spare”  and  “Replace  on-board  LRU  with  LRU  cannibalized  from 
another  shop”  most,  by  937%,  and  decreases  the  return  of  Process  6,  “NME  repairs  on¬ 
board  LRU  and  returns  it  to  ship,”  by  73%. 

Comparison  of  Royal  Dutch  Navy  and  U.S.  Navy  Scenarios 

Previous  research  using  the  KVA  approach  developed  estimates  of  returns  on 
technology  investment  of  a  scenario  in  which  the  U.S.  Navy  adopts  3D  LST  and  CPLM  tools 
into  the  SHIPMAIN  program.  Komoroski  (2005)  investigated  the  early  phases  of  SHIPMAIN 
(see  Table  3).  Adding  the  3D  LST  and  CPLM  technologies  improves  the  overall  preparation 
for  maintenance  process  ROI.  Adding  these  technologies  generally  improves  individual 
processes  as  well.  The  range  of  improvements  across  individual  processes  is  large,  varying 
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from  0%  (Issue  Tasking)  to  3,031%  (Generate  Drawings).  Cost  reduction  explains  these 
differences.  For  example,  the  adoption  of  technology  in  Core  Processes  4  (Conduct 
Shipcheck)  and  7  (Generate  Drawings)  would  significantly  reduce  the  number  of  people 
required  to  survey  ship  conditions  (4)  or  draft  3D  drawings  from  the  survey  data  (9),  resulting 
in  large  ROI  if  the  technology  is  adopted. 

Seaman,  Housel,  and  Mun  (2007)  used  KVA  to  model  the  later  phases  of  SHIPMAIN 
(see  Table  3).  Adding  the  technologies  also  improves  overall  maintenance  implementation 
process  ROI.  Adding  these  technologies  also  improves  each  of  the  individual  processes. 

The  range  of  improvements  across  individual  processes  is  large,  varying  from  6%  to  466% 
(Final  Install,  Closeout  SC),  although  not  as  wide  as  in  the  preparation  for  maintenance 
processes  (see  Seaman,  Housel,  &  Mun,  2007,  for  details). 

Table  3.  Return  on  Investment:  Baseline  and  Technology  Adoption  Scenarios 


Baseline 
Overall  ROI 

Technology- 
adopted 
Overall  ROI 

US  Navy -SHIPMAIN 

(preparation  for 
maintenance  phases) 

-27% 

2019% 

US  Navy -SHIPMAIN 

(implementation  phases) 

35% 

201% 

Royal  Dutch  Navy 

(Damen  experience 
extrapolation) 

35% 

274% 

The  three  scenarios  have  some  similarities.  For  all  three,  overall  ROIs  after 
technology  adoption  are  positive  and  large.  This  supports  the  adoption  of  advanced 
technologies,  such  as  3D  LST,  3D  PDF  models,  and  CPLM,  to  improve  the  efficiency  of 
resource  use.  The  scenarios  also  have  potentially  significant  differences.  The  technology 
adoption  scenario  for  the  preparation  for  maintenance  phases  of  the  U.S.  scenario  has  a 
much  higher  overall  ROI  than  the  ROIs  for  the  maintenance  implementation  phases  of  the 
U.S.  or  the  Dutch  scenario  (2,019%  »  201%  or  274%).  Several  factors  could  explain  these 
differences. 

•  The  preparation  for  maintenance  phases  of  the  U.S.  scenario  have 
significantly  lower  ROI  in  the  As-ls  (without  technology)  condition  (-27%  > 
35%).  This  suggests  that  inefficiencies  in  the  preparation  for  maintenance 
processes  provided  more  and  larger  opportunities  for  improvement. 

•  The  individual  preparation  for  maintenance  processes  that  increased  the 
most,  such  as  Generate  Drawings  and  Conduct  Shipcheck,  are  very  labor 
intensive  and,  therefore,  costly,  providing  large  opportunities  for  cost 
reduction  through  technology  adoption. 

•  Several  of  the  individual  maintenance  implementation  processes  are  labor 
intensive  but  less  impacted  by  technology  (e.g.,  Install  Shipcheck),  thereby 
making  those  changes  in  ROI  less  dramatic. 

•  The  preparation  for  maintenance  phases  of  the  U.S.  scenario  could  be  more 
optimistic  in  their  projections  than  the  other  scenarios. 

•  The  estimates  of  process  changes  may  use  different  assumptions. 
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•  Technologies  adopted  in  the  preparation  for  maintenance  phases  of  the  U.S. 
scenario  may  make  much  larger  improvements  in  processes  than  those  in  the 
maintenance  implementation  phases  of  the  U.S.  or  the  Dutch  scenario. 

•  The  Dutch  case  does  not  use  all  of  the  capabilities  of  the  CPLM,  thereby 
making  it  more  incremental  than  the  U.S.  scenarios,  in  which  all  the 
capabilities  of  the  CPLM  were  projected  to  be  used.  Also,  3D  PDF  has  more 
limited  capabilities  for  integration  with  the  CPLM  logistics  package  when 
compared  to  the  integration  of  3D  LST  capabilities  for  broader  usage  in 
requirements  analysis,  planning  for  maintenance,  and  tracking  of  parts  in  the 
supply  chain  and  across  suppliers  and  contractors.  This  can  partially  explain 
the  lower  ROI  for  the  Dutch  technology-adopted  scenario  than  for  the  U.S. 
preparation  for  maintenance  scenario. 

•  The  projections  of  the  impacts  on  the  maintenance  implementation  phases  of 
the  U.S.  scenario  and  the  Dutch  scenario  may  be  rather  conservative  based 
on  research  into  the  actual  successful  implementation  of  other  modern 
technologies,  such  as  RFID  in  inventory  management.  In  a  study  of  the  actual 
use  of  passive  RFID  in  two  military  warehouses  in  the  Korean  air  force  and 
army,  the  actual  ROIs  from  use  of  the  RFID  technology  were  more  than  triple 
the  projected  impact  of  the  use  of  the  technology  in  a  separate  study  of  the 
U.S.  Navy  (Courtney,  1997).  The  Korean  ROIs  after  actual  implementation  of 
the  RFID  technology  ranged  from  610%  to  576%,  compared  to  the  projected 
returns  anticipated  from  the  implementation  of  the  same  technology  in  the 
U.S.  Navy,  which  ranged  up  to  133%.  The  implication  is  that  actual 
successful  implementation  of  information  technology  in  a  military  may  exceed 
projections  of  the  potential  impacts  of  the  technology.  It  follows  that  the 
current  research  on  the  impacts  of  CPLM  and  3D  LST  or  3D  PDF  may  be 
more  conservative  than  the  reality  once  these  technologies  are  actually 
implemented  on  a  wide-scale  basis. 

Integrated  Risk  Management  Modeling  and  Results 

Through  the  use  of  Monte  Carlo  simulation,  the  resulting  stochastic  KVA  ROK  model 
yielded  a  distribution  of  values  rather  than  a  point  solution.  Thus,  simulation  models  analyze 
and  quantify  the  various  risks  and  uncertainties  of  each  program.  The  result  is  a  distribution 
of  the  ROKs  and  a  representation  of  the  project’s  volatility. 

It  is  important  to  understand  why  it  is  necessary  to  apply  uncertainty  to  the  model. 
Because  the  KVA  process  provided  a  point  value  for  each  quantity,  even  though  there  was 
some  uncertainty  in  the  estimates  provided  by  the  subject-matter  experts,  application  of  the 
appropriate  statistical  distributions  of  input  was  used  to  restore  the  real  world’s  uncertainty 
to  the  model.  Having  inputs  from  only  three  experts,  as  opposed  to  hundreds  of  estimates, 
and  rather  than  using  these  three  discrete  inputs,  we  applied  the  lessons  learned  in  cost 
estimating  as  reflected  in  the  Air  Force  handbook  (U.S.  Air  Force,  2007)  as  a  good  starting 
point  for  representing  the  uncertainty  and  reflecting  it  in  the  simulations. 

Next,  using  the  developed  KVA  model,  risk  simulation  probabilistic  distributional  input 
parameters  were  inserted  into  the  three  main  variables:  percentage  automation,  time 
process  is  executed,  and  average  time  to  complete.  A  risk  simulation  of  10,000-1,000,000 
simulation  trials  was  run  to  obtain  the  results. 

At  this  point  in  the  analysis,  a  proxy  for  revenues  and  volatility  has  been  identified,  as 
well  as  the  numerators  and  denominators  for  the  ship  maintenance  program.  The  next  step 
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is  to  define  or  frame  the  alternatives  and  approaches  to  implementing  3D  PDF  and  Logistics 
Team  Centers,  namely,  strategic  real  options.  The  questions  that  can  be  answered  include 
the  following:  What  are  the  options  involved?  How  should  these  new  processes  be  best 
implemented?  Which  decision  pathway  is  optimal?  and  How  much  is  the  program  worth  to 
the  DoD? 

Integrated  Risk  Management:  Framing  the  Real  Options 

As  part  of  the  first  round  of  analysis,  Figure  3  illustrates  some  of  the  potential 
implementation  paths  for  3D  PDF/Logistics  TC.  Clearly,  some  of  the  pathways  and  flexibility 
strategies  can  be  refined  and  updated  through  the  passage  of  time,  actions,  and  events. 
With  the  evolution  of  the  implementation,  valuable  information  is  obtained  to  help  in  further 
fine-tuning  the  implementation  and  decision  paths. 

For  the  preliminary  analysis,  the  following  options  were  identified,  subject  to 
modification: 

•  Option  A:  As-ls  Base  Case.  The  ROI  for  this  strategic  path  is  computed  using 
the  baseline  KVA  and  this  represents  the  current  RDN  ship  maintenance 
process  (i.e. ,  no  newly  added  technologies). 

•  Option  B:  Execute  and  implement  3D  PDF  and  Logistics  package 
immediately  across  all  RDN  ship  maintenance  processes.  That  is,  take  the 
risk  and  execute  on  a  larger  scale,  where  you  would  spend  the  initial 
investments  and  continuing  maintenance  expenses  required  and  take  on  the 
risks  of  any  potential  failure,  but  reap  the  rewards  of  the  new  processes’ 
savings  quickly  and  immediately.  The  analysis  is  represented  as  the  current 
RDN  process  altered  to  reflect  what  we  estimate  to  be  the  impacts  of 
adopting  both  a  Logistics  package  and  3D  PDF  models. 

•  Option  C:  This  represents  the  current  RDN  process  altered  to  reflect  what  we 
estimate  to  be  the  impacts  of  adopting  3D  PDF  models  and  managing  them 
in  a  Team  Center  or  similar  product.  This  technology  was  chosen  largely 
because  Damen  is  developing  and  pursuing  the  use  of  this  technology. 

•  Option  D:  This  implementation  pathway  represents  the  current  RDN  process 
altered  to  reflect  what  we  estimate  to  be  the  impacts  of  managing  using  a 
Logistics  module  in  a  Team  Center  or  similar  product.  This  technology  was 
chosen  partially  because  it  was  a  technology  that  Damen  considered,  but 
chose  not  to  purchase. 

•  Option  E:  Proof  of  Concept  (POC)  approach.  That  is,  to  execute  large-scale 
implementation  of  3D  PDF  and  Logistics  Module  in  TC  only  after  an  initial 
POC  shows  promising  results.  If  POC  turns  out  to  be  a  failure,  we  walk  away 
and  exit  the  program,  and  losses  are  minimized  and  limited  to  the  initial  POC 
expenses.  Proceed  to  full  implementation  in  POC  programs  first  and  then 
expand  in  sequential  fashion  to  other  programs,  based  on  where  best  ROI 
estimates  are  shown. 

•  Option  F:  POC  on  3D  PDF  only.  Assuming  the  POC  works  and  3D  PDF  is 
executed  within  a  few  programs  successfully,  the  learning  and  experience 
obtained  becomes  valuable  and  allows  the  shipyard  to  expand  its  use  into 
many  other  programs  or  perhaps  across  the  RDN. 

•  Option  G:  POC  on  Logistics  Module  in  TC  only.  Assuming  the  POC  works 
and  Logistics  Module  is  executed  within  a  few  programs  successfully,  the 
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learning  and  experience  obtained  becomes  valuable  and  allows  the  shipyard 
to  expand  its  use  into  many  other  programs  or  perhaps  across  the  RDN. 


AS-tS 


Strategy* 


As-ls  Base  Case  The  ROMs  strategic  path  is  computed  using  the  Paseline  KVA  and  this 
represents  the  current  Royal  Dutch  Nay  ship  maintenance  process  -  i.e.  no  added  technologies 


Strategy  B 


Strategy  D 


Start 

■a 


Sr 


Strategy  E 


Strategy  F 


Strategy  6 


30  Only 


3D  PDF  &  Logistics 


Implement  Now 


30  PDF  Only 


Strategy  C 

Logistics  Only 


Implement  Now 


30  &  Logistics 


Implement  Now 


Execute  and  implement  30  PDF  and  Logistics  package  immediately  across  all  Royal  Dutch  Navy  ship  maintenance  processes  That  is.  take  he 
nsk  and  erectile  on  a  larger  scale,  where  you  would  spend  the  initial  investments  and  continuing  maintenance  expenses  required  and  lake  on 
he  risks  ot  any  polental  failure  Put  reap  he  rewards  othe  new  processes'  savings  quickly  and  immediately  The  analysis  is  represented  as  he 
current  RDN  process  altered  to  retied  what  we  estimate  to  Pe  he  impacts  of  adopting  both  a  Logistics  package  and  3D  PDF  models 

This  represents  he  current  RDN  process  altered  to  reflect  what  we  esf  mate  to  Pe 
he  impacts  ef  adopting  3D  PDF  models  and  managing  hem  in  a  Team  Center  or 
similar  product  This  technology  was  chosen  largely  because  Damen  Is 
developing  and  pursuing  he  use  of  his  technology. 


This  implementation  pathway  represents  he  current 
RDN  process  altered  to  refled  what  we  estimate  to  be 
he  impacts  ot  managing  using  a  logistics  module  in 
a  Team  Center  or  similar  product  This  technology 
was  chosen  paitialty  because  it  was  a  technology  hat 
Damen  considered  but  chose  not  to  purchase 

3D  Implementation 


Logistics  Implementation 


Pure  II 


Exit 

Exit  after  Phase! 


Small  Implementation 


_  ^  Exit 

Exit  after  Phase  I 


Full  Implementation 


Sr 


Exit 

Exit  after  Phase  l 


Small  Implementation 


a  Exit 

^  Exit  after  Phase  I 


Full  Implementation 


Phut  111 


Logistics  Only 


Exit 

EnfafttrPfcaMl 


Prool  ot  Concept  approach,  hat  is,  to  execute  large-scale  implementation  ol  30  PDF 
and  Logistics  Module  in  TC  only  alter  an  Initial  Prool  ol  Concept  (POC)  shows 
promising  results.  It  POC  turns  out  to  be  a  failure,  we  walk  away  and  exit  he 
program,  and  losses  are  minimiad  and  limited  to  he  initial  POC  expenses.  Pioceed 
to  full  implementation  in  POC  programs  first  and  then  expand  in  sequential  fashion 
to  other  programs,  Cased  on  where  cost  ROI  estimates  are  shown. 


Prool  ot  Concept  on  3D  PDF  only  Assuming  he  POC  works 
and  30  PDF  is  executed  within  a  tew  programs  successfully, 
he  learning  and  expenence  obtained  becomes  valuable  and 
allows  he  shipyards  to  expand  its  use  into  many  other 
programs  or  perhaps  across  he  Royal  Dutch  Navy. 


^  Exit 

Exit  after  Phase  l 


Prool  of  Concept  on  Logistics  Module  in  TC  only  Assuming  he  POC 
works  and  Logistics  Module  is  executed  within  a  tew  programs 
successfully,  he  learning  and  expenence  obtained  becomes  valuable 
and  allows  he  shipyards  to  expand  its  use  into  many  oher  programs  or 
perhaps  aaoss  he  Royal  Dutch  Navy. 


Figure  3.  Sample  Real  Options  Values 

Integrated  Risk  Management:  Strategic  Flexibility  Real  Options  Results 

Figure  4  shows  the  results  of  the  strategic  real  options  flexibility  values  and 
compares  them  against  the  KVA  ROI  values.  Options  B  ($154.1  million  at  278%  ROI)  and  E 
($156.5  million  at  282%  ROI)  of  implementing  both  3D  PDF  and  Logistics  Module  TC  return 
the  highest  ROI  and  total  strategic  value,  and  both  provide  a  significant  value-add  above 
and  beyond  Option  A’s  As-ls  condition  ($31 .9  million  at  35%  ROI).  As  Options  B  and  E  are 
the  most  significant;  stage-gating  the  implementation  over  several  phases  yields  a  slightly 
higher  value  (Option  E  exceeds  Option  B  by  about  $2.4  million). 

In  addition,  the  Monte  Carlo  risk  simulation  results  on  the  real  options  values  were 
developed  (but  are  not  shown  here  for  brevity;  see  Ford,  Housel,  &  Mun,  2012).  In 
comparing  Options  E  and  F,  there  is  a  94%  probability  that  Option  E,  which  has  a 
sequentially  phased  implementation  of  both  3D  PDF  and  Logistics  Module  TC,  provides  a 
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better  return  than  Option  F.  In  comparing  Options  E  and  B,  there  is  a  95%  confidence  that, 
even  with  all  the  uncertainties  in  the  collected  data  and  risks  of  implementation  success, 
including  uncertainties  of  whether  the  estimated  returns  will  materialize  and  so  forth,  there  is 
at  least  a  $1.27  million  net  advantage  in  going  with  Option  E.  Therefore,  it  is  better  to 
sequentially  phase  and  stage-gate  the  implementation  over  several  years,  allowing  the 
ability  to  exit  and  abandon  further  stages  if  events  unfold  and  uncertainties  become 
resolved,  so  that  further  investment  in  the  technology  no  longer  makes  sense.  The  risk- 
simulated  real  options  value  has  an  expected  value  (mean)  of  $195  million,  with  a 
corresponding  average  ROI  of  363%. 


ANALYSIS  RESULTS 
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Figure  4.  Sample  Real  Options  Values 

Summary  Results  of  the  Integrated  Risk  Management  Analysis 

IRM  and  strategic  real  options  methodologies  were  applied  to  the  KVA-SD  results, 
and  the  results  indicate  that  Option  B  had  a  value  of  $154.1  million  (278%  ROI)  and  Option 
E  had  a  value  of  $156.5  million  (282%  ROI),  where  both  options  indicate  that  implementing 
3D  PDF  and  Logistics  Module  TC  return  the  highest  ROI  and  total  strategic  value  and  both 
provide  a  significant  value-add  above  and  beyond  Option  A’s  As-ls  condition,  with  a  value  of 
$31 .9  million  (35%  ROI).  As  Options  B  and  E  are  most  significant,  we  know  that 
implementation  of  3D  PDF  and  Logistics  Module  TC  returns  the  highest  value  and,  when 
implemented  over  time  in  a  stage-gate  process  over  several  phases,  would  yield  a  slightly 
higher  value  (Option  E  exceeds  Option  B  by  about  $2.4  million).  Therefore,  we  conclude  that 
3D  PDF  and  Logistics  Module  TC  implemented  in  a  phased  stage-gate  environment  would 
yield  the  best  results.  In  comparing  Options  E  and  B,  there  is  a  95%  probability,  even  with  all 
the  uncertainties  in  the  collected  data  and  risks  of  implementation  success  as  well  as  the 
uncertainties  of  whether  the  estimated  returns  will  materialize,  that  there  is  a  $1 .27  million 
net  advantage  in  going  with  Option  E  to  sequentially  phase  and  stage-gate  the 
implementation  over  several  years,  allowing  the  ability  to  exit  and  abandon  further  stages  if 
events  unfold  and  uncertainties  become  resolved  so  that  further  investment  in  the 
technology  no  longer  makes  sense. 

Conclusions 

We  collected  new  data  on  ship  maintenance  processes  and  the  use  and  adoption  of 
technologies  in  ship  maintenance  by  the  RDN  and  Damen  Shipbuilding.  The  data  were  used 
to  build  and  calibrate  a  system  dynamics  model  of  RDN  ship  maintenance.  Model 
simulations  of  four  technology  adoption  scenarios,  reflecting  the  use  of  two  available  or 
developing  technologies,  generated  estimates  of  maintenance  operations  behavior  that 
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were  imported  into  KVA  models.  The  four  technology  adoption  scenarios  were  then  modeled 
in  the  KVA  models.  The  KVA  models  estimated  the  ROIs  for  individual  processes  and  ship 
maintenance  as  a  whole  for  each  scenario.  Results  were  analyzed  to  reveal  the  relative 
improvement  provided  by  individual,  and  combinations  of,  technologies. 

The  results  of  this  study,  in  combination  with  prior  studies,  make  it  evident  that  the 
technologies  under  review  will  make  large  contributions  to  cost  reductions  in  ship 
maintenance  processes.  These  conclusions  are  supported  by  the  comparative  analysis  of 
the  Dutch  experience  with  similar  supporting  technologies.  There  appears  to  be  no  empirical 
evidence  that  would  serve  as  an  impediment  to  adopting  the  technologies  in  the  near  term 
rather  than  the  longer  term.  We  recommend  an  immediate  adoption  of  the  3D  LST  and 
CPLM  technologies  to  support  ship  maintenance  processes. 

Implications  for  Acquisition  Practice 

The  current  research  has  significant  implications  for  acquisition  practice.  First,  the 
conclusions  support  multiple  previous  investigations  that  recommend  the  adoption  of 
available  information  technologies  to  reduce  the  costs  of  U.S.  Navy  ship  maintenance. 
Second,  multiple  significantly  different  technologies  (e.g.,  3D  LST,  3D  PDF,  logistics 
support)  can  improve  ship  maintenance  operations.  Third,  among  those  studied,  the 
expensive  information  technologies  were  found  to  benefit  high-cost  processes  the  most, 
such  as  where  labor  can  be  replaced  with  technology.  Doing  so  reduces  costs  and 
increases  production  rates  by  reducing  cycle  times.  This  implies  that  if  technology  adoption 
efforts  are  to  be  prioritized,  those  with  labor-intensive  processes  that  can  be  replaced  with 
technology  should  be  given  higher  priority.  The  real  options  analysis  of  implementation 
strategies  demonstrated  that  some  technologies  (3D  PDF  in  this  case)  can  dominate  the 
value  space  and  that  phased  implementation  adds  value  compared  to  one-step 
implementation.  The  results  of  the  current  work  recommend  a  careful  investigation  of 
available  technologies  and  how  they  improve  operations,  followed  by  a  phased  development 
and  implementation  of  the  adoption  of  the  chosen  technologies. 

Implications  for  Research 

The  results  of  the  three  KVA-based  studies  varied  significantly.  A  likely  cause  is  the 
difficulty  in  accurately  forecasting,  in  quantitative  terms,  the  impacts  of  new  technologies  on 
specific  processes.  The  use  of  data  and  information  from  organizations  that  are  actively 
developing  and  adopting  information  technologies  (Damen)  and  performing  operations 
similar  to  those  performed  by  the  U.S.  Navy  (RDN)  proved  to  be  very  valuable  in  improving 
the  models  (e.g.,  by  adding  the  3D  PDF  technology).  Therefore,  further  refinement  of  the 
models  should  include  actual  application  data,  such  as  a  study  of  actual  technology 
adoptions  by  the  U.S.  Navy. 
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Abstract 

With  few  exceptions,  studies  on  improving  the  acquisition  of  weapon  systems  and  services 
within  the  DoD  observe  that  the  process  takes  too  long.  A  2010  report  of  a  study  led  by 
former  Secretary  of  Defense  William  Perry  and  former  Assistant  to  the  President  for  National 
Security  Affairs  Stephen  Hadley  entitled  The  QDR  in  Perspective:  Meeting  America’s 
National  Security  Needs  in  the  21st  Century:  The  Final  Report  of  the  Quadrennial  Defense 
Review  Independent  Panel  supported  this  point  of  view,  asserting  that  no  defense  program 
should  exceed  seven  years.  In  a  September  14,  2010,  memorandum,  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  called  for  the  DoD  acquisition  community 
to  “set  shorter  program  timelines  and  manage  to  them.”  But  what  is  the  right  timeline  for  a 
given  defense  program?  The  author  offers  a  methodology  for  making  that  determination 
through  a  process  using  time  as  an  independent  variable  (TAIV™)1  in  a  way  similar  to  using 
cost  as  an  independent  variable  (CAIV).  Using  TAIV™  establishes  a  credible  way  of 
reconciling  cost,  capability,  and  the  time  required  to  field  a  needed  capability. 

Introduction 

Although  the  work  done  in  assessing  the  value  of  using  time  as  an  independent 
variable  (TAIV™)  is  not  a  panacea,  there  are  clearly  acquisition  programs  where  cost  and 
performance  should  vary  with  program  management  conditions  that  recommend  accepting 
more  performance  at  an  increase  in  cost.  What  the  National  Defense  Business  Institute 
(NDBI)  has  found  is  that  assessing  a  capability  relative  to  the  time  necessary  to  achieve  that 
capability  is  a  useful  effort.  TAIV™  can  be  a  valuable  tool  to  that  end. 

From  the  end  of  the  Korean  War  to  the  present,  the  length  of  time  required  to  field  a 
Major  Defense  Acquisition  Program  (MDAP)  has  persistently  grown  as  a  common  practice. 

In  their  assessment  Streamlining  DoD  Acquisition:  Balancing  Schedule  With  Complexity, 
James  Rothenflue  and  Marsh  Kwolek  (2010)  put  it  more  bluntly: 

Since  at  least  the  late  1960s,  the  Department  of  Defense  has  been  trapped  in 
an  escalating  cycle  of  cost  overruns  and  schedule  delays  on  large  acquisition 
programs.  In  particular,  state-of-the-art  aircraft  programs  have  ballooned  from 
one  to  five  year  sprints  during  and  immediately  after  World  War  II  to  the  25- 
year  marathons  of  the  present  day. 

With  the  ever-increasing  length  of  time  taken  to  field  weapon  systems,  total  program  costs 
have  risen  as  well. 


^AIV™  is  an  acronym  trademarked  by  Monitor  Government  Venture  Service,  LLC,  during  the  course 
of  their  research  and  analysis  of  the  use  of  time  as  an  independent  variable,  sponsored  by  the  Office 
of  the  Deputy  Secretary  of  Defense. 
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Analysis  over  time  shows  that  acquisition  programs  generally  grow  about  50%  in 
cost  (Younossi  et  al.,  2006),  and  of  course  larger  defense  programs  have  higher  stakes 
owing  to  the  sums  of  money  involved  compared  with  programs  managed  by  other  federal 
agencies.  Programs  with  longer  timeframes  for  engineering,  manufacturing,  and 
development  also  experience  greater  cost  growth  (Younossi  et  al.,  2006).  Furthermore, 
almost  all  acquisition  strategies  lack  the  analysis  as  to  what  time  does,  as  an  independent 
variable,  to  the  trade  space  defined  by  the  minimum  and  optimum  performance  and  cost. 

Two  recent  commentaries  on  the  crucial  nature  of  time  as  a  key  element  in  acquiring 
weapons  and  military  equipment  come  from  different  quarters,  but  they  agree  on  the  way 
forward.  In  a  2010  study,  two  former  senior  government  officials  argued  that  study,  which 
was  led  by  former  Secretary  of  Defense  William  Perry  and  former  Assistant  to  the  President 
for  National  Security  Affairs  Stephen  Hadley  entitled  The  QDR  in  Perspective:  Meeting 
America’s  National  Security  Needs  In  the  21st  Century:  The  Final  Report  of  the  Quadrennial 
Defense  Review  Independent  Panel.  Hadley  and  Perry  (2010)  were  quite  clear  in  their 
recommendation,  explaining, 

Permitting  delivery  times  longer  than  a  reasonably  achievable  standard  is 
counterproductive  to  both  the  demand  for  responsiveness  to  current  needs 
and  tomorrow's  challenges.  For  major  programs  for  future  forces,  useful 
increments  of  military  capability  should  be  defined  as  what  can  be  delivered 
within  5  to  7  years  with  no  more  than  moderate  risk. 

Undersecretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  Dr.  Ashton 
Carter,  also  addressed  the  issue  of  time  in  a  2010  memorandum  to  acquisition  professionals 
titled  Better  Buying  Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in 
Defense  Spending.  As  one  of  his  23  principal  actions  “to  improve  efficiency”  in  the  DoD 
acquisition  efforts,  Dr.  Carter  mandated,  “Set  shorter  program  timelines  and  manage  to 
them.” 


The  significance  of  time  to  more  efficient  acquisition  of  military  weapons  and 
equipment  is  not  new.  The  Special  Assistant  to  the  Deputy  Secretary  of  Defense  in  2004 
asked  the  Monitor  Group  (Monitor  Company  Group  LP,  2003)  to  look  at  the  value  of 
establishing  time  as  a  boundary  condition  or  driver  in  determining  the  desired  timeframe 
between  Milestone  B  and  initial  operating  capability.  Time  should  be  considered  an 
independent  variable  (TAIV™),  especially  when  it  is  critical  to  field  a  capability  on  a  specific 
point  in  the  future  to  have  a  positive  impact  on  a  threat  or  in  the  course  of  ongoing  combat. 
Because  there  is  little  direct  research  on  TAIV™  specifically,  the  Monitor  Group  work  done 
for  the  Office  of  the  Deputy  Secretary  of  Defense  and  subsequent  analyses  completed  for 
the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (2006) 
establishes  most  of  the  foundational  thinking  on  using  time  to  establish  time  as  a  structured 
way  to  determine  the  limits  or  boundaries  of  acquisition  programs.  However,  it  is  equally 
important  that  there  be  a  reliable,  valid  process  for  the  DoD  to  “evaluate,  acquire  and 
deploy”  the  most  current  and  effective  technology  for  “long-term  performance  and  mission 
accomplishment”  (Sherman  &  Rhoades,  2010). 

Unfortunately,  the  idea  did  not  gain  traction  in  2004,  for  two  related  reasons.  Among 
the  acquisition  community,  there  is  a  pervasive  belief  that  a  capability  is  selected  to  address 
a  requirement  that  then  takes  as  long  as  it  takes.  Such  an  attitude  carries  with  it  no 
discipline  or  structure  and  allows  for  program  success  to  be  dependent  on  yet-to-be-realized 
inventions  and  even  miracles.  That  lack  of  discipline  creates  a  work  environment  governed 
by  manufactured  job  security  rather  than  efficiency.  By  allowing  programs  to  be  temporally 
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indeterminate,  government  employees  and  contractors  have  no  incentive  to  bring  projects  to 
closure,  allowing  programs  to  drag  out  for  longer  periods  of  time. 

Times  have  changed,  and  the  pressure  is  intense  to  reduce  the  time  it  takes  to  put 
weapon  systems  and  equipment  in  the  field  and  simultaneously  reduce  the  costs  that  attend 
prolonged  and  stretched  programs.  The  time  is  opportune  for  applying  the  TAIV™  tool  for 
new  acquisition  programs,  as  well  as  to  block  upgrades  to  existing  programs. 

Applying  TAIV™ 

How  should  TAIV™  work  in  a  practical  sense?  When  the  Monitor  Group  in  2004 
attempted  to  use  time  to  drive  discipline  and  structure  into  the  acquisition  system,  the  idea 
was  to  use  TAIV™  to  enable  the  DoD’s  transformation  initiatives  as  a  supporting  acquisition 
framework.  There  was  sense  of  urgency  in  getting  needed  weapon  systems  and  equipment 
to  the  warfighters  in  both  Iraq  and  Afghanistan. 

If  TAIV™  is  to  be  used  and  applied  to  the  full  program  life-cycle,  then  the  TAIV™ 
analysis  must  start  early  in  the  acquisition  program  concept  development  and  acquisition 
process.  TAIV™  must  be  an  essential  part  of  the  acquisition  strategy,  solicitation  process, 
and  contract  development  activities. 

The  DoD  generally  pursues  one  of  three  approaches  when  acquiring  a  weapon 
system  or  piece  of  equipment:  single  step  to  full  capability,  incremental  development,  or 
spiral  development.  Incremental  and  spiral  development  are  grouped  under  the  category  of 
evolutionary  acquisition  strategy  (DoD,  2011).  The  acquisition  approach  must  consider  what 
the  desired  end  state  of  the  program  is  to  be  and  determine  the  appropriateness  TAIV™. 
Single-step  to  full  capability  can  apply  TAIV™  successfully  when  the  end-state  requirements 
are  known  and  can  be  used  to  set  program  length,  program  milestones,  and  incentivize 
compliance.  Additionally,  the  technology  must  be  mature.  Commodity  parts  and  immediately 
available  capability  would  fit  the  single-step  to  full-capability  category. 

Applying  TAIV™  to  an  incremental  development  approach  fits  when  the  end-state 
requirements  are  understood  and  when  multiple  development  cycles  are  anticipated.  Again, 
mature  technologies  are  required,  as  well  as  threats  that  can  be  addressed  with  minimum 
rather  than  assured  operating  capability.  TAIV™  would  be  used  to  set  the  increment 
duration  and  would  be  useful  as  an  incentive  to  drive  compliance. 

The  last  acquisition  approach,  spiral  development,  is  most  appropriate  when  multiple 
development  cycles  are  anticipated  and  the  acquisition  program  will  produce  interim 
outputs.  End-state  requirements  may  not  be  certain.  Programs  in  this  category  may  have 
some  level  of  exploratory  development,  but  with  mature  technology.  To  address  threats 
effectively,  sufficient  capability  is  required  sooner,  rather  than  assured  or  objective 
performance  capability  later. 

Generally,  when  time  is  the  subject  of  research,  it  is  used  in  terms  of  reducing  cycle 
time  for  acquiring  systems  and  equipment,  or  in  other  words,  looking  at  the  time  from 
program  start  to  delivery  from  the  top  down.  The  intent  is  to  simply  compress  or  streamline 
the  acquisition  process  and  drive  time  out  with  “levers  for  reducing  cycle  time”  (Sherman  & 
Rhoades,  2010).  The  traditional  approach  emphasizes  ways  to  reduce  time  spent  on 
activities  or  events  that  are  in  progress.  TAIV™  takes  an  alternative  approach  by 
establishing,  from  the  outset,  what  performance  or  capability  is  possible  based  on  the  time- 
defined  construct,  or  when  the  weapon  system  or  equipment  must  be  in  the  field.  Once  the 
capability  available  to  the  time-defined  limit  is  determined,  then  the  program  will  be  driven  to 
that  boundary. 
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The  green  line  in  Figure  1  represents  the  sequence  of  technology  maturation  over 
time.  The  TAIV™  process  looks  at  the  maturity  and  relevance  of  a  necessary  technology 
that  provides  a  warfighting  capability  starting  at  the  beginning  of  the  technology  maturation 
line  (point  1).  The  distance  between  points  1  and  2  represents  the  time  necessary  to 
develop,  adapt,  or  exploit  a  maturing  technology  and  turn  it  into  a  capability.  Point  2  is  the 
best  time-to-field  versus  capability  increase.  It  is  at  this  point  that  the  maximum  amount  of 
capability  for  the  technology  available  is  realized.  The  timeframe  represented  by  point  3 
allows  for  fielding  and  follow-on  production  of  a  capability.  Taking  additional  time  to  develop 
a  particular  technology  will  not  provide  marginally  greater  capability  until  point  4,  when  there 
is  another  technology  breakthrough.  As  demonstrated,  TAIV™  puts  the  emphasis  on  fielding 
capability  when  the  technology  supporting  the  capability  has  its  greatest  value.  That  value  is 
defined  by  the  lack  of  an  equal  alternative  technology  that  meets  the  time-defined  capability. 


Figure  1.  Time  as  an  Independent  Variable  (TAIV™)  for  Fielding  Capability 

Note.  Identifying  the  “Critical  Time  vs.  Capability  Point”  where  extending  the  time  does  not  achieve  a 
marginally  greater  degree  of  capability  is  the  analytic  value  of  TAIV™ . 


Coincidentally,  the  technologies  that  underpin  the  capabilities  that  might  be  needed 
in  the  future  continue  to  mature.  Overtime,  significant  breakthroughs  occur,  increasing  the 
potential  for  ever-greater  capability,  but  only  after  a  capability  has  been  fielded.  When 
additional  time  spent  on  development  of  technology  no  longer  increases  the  level  of 
capability,  that  is  the  critical  time  to  field  a  capability.  The  existing  technology  should  be 
exploited  from  that  point  until  there  is  another  technology  breakthrough  or  dramatic 
technology  increase. 
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Should  the  need  for  a  capability  be  more  urgent  because  of  a  more  near-term  threat, 
the  level  of  technology  may  have  to  be  less  capable.  In  such  a  case,  an  appropriate  strategy 
would  be  single-step  to  full  capability.  Commercial  off-the-shelf  (COTS)  technology  provides 
a  solution  with  little  or  no  need  for  integration  or  development  time  and  would  aid  in  putting  a 
capability  in  the  field  in  the  least  amount  of  time.  The  concept  of  using  time  to  drive  the  level 
of  technology  that  defines  a  capability  can  be  used  to  assess  the  appropriate  capability  and 
performance  trades. 

The  challenge  for  the  acquisition  community  is  twofold.  First,  the  point  in  time  must 
be  established  at  which  a  weapon  system  should  be  fielded.  Second,  the  available  COTS 
capability  must  be  determined  in  terms  of  what  can  be  most  easily  developed.  As  Figure  1 
shows,  getting  greater  capability  to  the  field  earlier  relies  on  exploiting  what  is  available  that 
requires  little  or  no  development.  The  ideal  condition  achieves  operational  capability  at  the 
point  where  additional  time  does  not  gain  appreciably  greater  capability — the  critical  time 
versus  capability  point,  or  the  knee  in  the  TAIV™  curve. 

The  crux  of  the  TAIV™  tool’s  value  comes  from  its  ability  to  reveal  the  amount  of 
time  necessary  to  meet  a  required  fielding  date  with  the  most  capability. 

As  depicted  in  Figure  1,  the  TAIV™  process  satisfies  all  three  of  the  acquisition 
approaches  described  previously.  Single-step  to  full  capability  can  be  achieved  when  the 
end-state  requirements  are  certain;  incremental  development  using  TAIV™  when  both  end- 
state  requirements  and  multiple  development  cycles  are  criteria;  and  a  spiral  development 
approach  would  benefit  from  TAIV™  when  multiple  development  cycles  and  interim  outputs 
are  anticipated. 

Figure  2  represents  how  threshold  and  objective  performance  parameters  can  frame 
the  trade  space  for  achieving  the  target  delivery  of  a  capability.  Again,  time  establishes  the 
boundaries. 


TAIV  Reveals  Trade  Space  Dimension 


Figure  2.  TAIV™  Reveals  Trade  Space  Dimension 

(adapted  from  2004  Monitor  Group  briefing) 

Note.  TAIV™  provides  a  means  of  identifying  the  best  value  performance  solution  to  field  an  effective 

capability  at  a  target  delivery  time. 
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Establishing  capability  and  the  desired  performance  with  objective  performance 
value  and  minimum  acceptable  or  threshold  performance  value  frame  the  capability 
performance  necessary  to  meet  a  requirement  and  address  a  threat.  The  trade  space 
becomes  the  time  between  the  earliest  that  the  threshold  performance  can  be  met  and  the 
latest  delivery  date  before  the  risk  of  significant  damage.  Again,  the  target  delivery  date  is 
defined  as  that  point  in  time  where  the  most  capability  or  performance  can  be  achieved, 
after  which  continued  pursuit  of  “better”  performance  has  little  or  no  increase  in  capability 
before  the  latest  delivery. 

The  target  delivery  date  is  also  the  point  in  the  future  when  the  best  capability  over 
time  can  be  achieved  at  the  best  value. 

It  is  a  reasonable  assumption  that  acquisition  programs,  guided  by  earliest  and  latest 
acceptable  time-to-field,  possess  a  credible  understanding  of  the  threat  to  be  addressed  by 
the  capability.  Here  again,  having  TAIV™  as  a  tool  lends  credibility  to  the  argument 
advocating  for  a  particular  capability  being  fielded  at  a  particular  point  in  future. 

TAIV™  Integrates  Threat  Assessment,  Time-to-Field  With  Available  Technology 

TAIV™  is  useful  in  helping  determine  the  greatest  capability  over  the  least  amount  of 
time,  but  some  forcing  function  must  be  present  to  actually  define  the  “least  amount  of  time.” 

At  minimum  there  must  be  some  understanding  of  the  driving  requirement  that 
addresses  an  understood  national  security  threat.  Then  “the  least  amount  of  time”  becomes 
the  time  to  field  the  weapon  systems  or  piece  of  equipment. 

Figure  3  illustrates  the  relationship  among  time,  capability,  and  technology  progress 
when  compared  to  the  window  of  understanding  represented  by  the  threat  a  potential 
enemy.  Over  time,  there  is  less  fidelity  in  the  understanding  of  threat  in  terms  of  the  lower 
and  upper  limits  of  the  capability  necessary  to  meet  a  requirement  to  address  the  threat. 

The  clearest  and  most  credible  understanding  of  the  threat  has  the  most  fidelity  in  the  near 
term.  Understanding  of  the  threat  declines  as  the  assessment  of  that  threat  is  pushed  further 
into  the  future. 
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Figure  3.  Capability  Over  Time  Relative  to  Level  of  Threat  Understanding 

(adapted  from  2004  Monitor  Group  briefing) 

Note.  As  planners  attempt  to  assess  the  threat  to  national  security,  there  is  less  fidelity  and  accuracy 
possible  the  further  into  the  future  the  assessment  is  made. 


Additionally,  as  Figure  3  suggests,  using  TAIV™,  it  is  more  likely  that  the  capability 
possible  in  the  near  term  (and  more  in  line  with  the  critical  time-to-field)  will  be  more 
appropriate  to  meeting  the  requirement  to  address  a  threat. 

When  TAIV™  is  applied  to  the  development  of  an  acquisition  program,  the 
importance  of  time  in  developing  and  defining  the  technology,  as  well  as  its  design  and 
production,  is  more  dominant  in  the  analysis  of  cost,  schedule,  and  achieving  the  desired 
performance.  Time-defined  in  this  instance,  however,  is  not  synonymous  with  schedule. 

Though  a  2009  GAO  report  points  out  that  a  DAPA  report  recommends  that 
schedule  be  a  key  performance  parameter  (KPP),  this  paper  looks  at  the  time  between  the 
beginning  of  a  program  and  the  point  at  which  a  weapon  system  is  operational  as  a  time- 
defined  period.  By  that  definition,  the  specific  length  of  time,  and  not  schedule,  is  a  KPP. 

Schedule  is  the  sequential  distribution  of  program  events  when,  on  completion,  they 
have  a  timeframe  associated  with  them.  We  measure  schedule  with  milestones 
accomplished.  TAIV™,  on  the  other  hand,  is  the  analytic  construct  that  identifies  which 
performance  capabilities  are  important  and  must  be  achieved,  and  conversely,  which  are  of 
marginal  value  when  considering  the  time  from  development  to  incorporation  into  the 
weapon  system  to  fielding  to  consideration  for  future  block  upgrades.  The  time-defined 
period  is  established  with  the  results  of  the  TAIV™  analysis. 

Various  recommendations  exist  for  the  timeframe  for  development  and  fielding: 
Hadley  and  Perry  (2010)  suggested  five  years  for  fighter  type  aircraft  and  eight  years  for 
ships,  and  five  to  seven  years  for  all  programs;  in  a  DAPA  report,  Kadish  (2006) 
recommended  “nominally  no  more  than  six  years  for  major  platforms  from  Milestone  A”  to 
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fielding.  Without  a  way  to  appropriately  evaluate  time,  acquisition  programs  will  continue  to 
take  “as  long  as  they  take.” 

Although  a  2009  GAO  report  points  out  that  the  DAPA  report  (Kadish,  2006) 
recommended  that  schedule  be  a  KPP,  this  paper  looks  at  the  time  between  the  beginning 
of  a  program  and  the  point  at  which  a  weapon  system  is  operational  as  a  time-defined 
period  and  that  specific  length  of  time  be  a  key  performance  parameter,  not  schedule. 

The  reason  for  distinguishing  between  time  and  schedule  is  to  give  precedence  to 
the  time-to-field  as  a  weapon  system  or  piece  of  equipment  that  should  take  precedence 
over  a  particular  sequence  of  program  events  or  activities.  The  integrated  master  schedule 
(IMS)  can  be  modified  during  the  course  of  program  duration  without  doing  violence  to  an 
established  critical  time  to  field  the  weapon  system  or  piece  of  equipment.  Establishing  a 
program  schedule  as  a  KPP  would  suggest  that  to  modify  the  schedule  while  not  changing 
cost  and  performance  or  the  time-to-field  would  constitute  a  failure  to  meet  a  KPP,  but  that 
would  not  necessarily  hold  true.  Making  schedule  a  KPP  simply  adds  a  level  of  complexity  to 
the  concept  of  TAIV™  without  corresponding  value. 

Urgency  for  fielding  a  particular  desired  capability,  then,  has  a  context  that  can  be 
used  to  describe  what  needs  to  be  fielded  or  deployed  and  when.  Again,  Kadish  (2006) 
fortified  this  line  of  thinking,  saying,  “Once  the  time-to-need  and  the  current  technology  risk 
level  are  determined  the  program  should  be  time-constrained.” 

The  TAIV™  curve  conjures  the  “cost  as  an  independent  variable”  curve  owing  to  the 
analogous  relationship  that  results  from  the  adage  “time  is  money.”  But  there  is  a  clear 
distinction:  cost  may  vary  with  time,  but  not  directly.  Many  variables  that  include  quantity  and 
quality  determine  cost,  but  time  is  immutable.  Just  as,  at  a  certain  point,  increasing  dollars 
spent  on  a  program  will  not  produce  a  corresponding  increase  in  capability,  increasing  time 
spent  will  also  not  produce  a  corresponding  and  direct  increase  in  capability.  Technology 
and  innovation  must  come  into  play.  Rene  Cordero  made  this  point  in  a  1991  discussion  of 
“managing  speed”  in  getting  products  to  market  to  avoid  obsolescence:  “Finally,  the  natural 
limits  of  the  technology  are  reached  and  only  small  improvements  of  the  product  are 
possible.”  That  point  in  Figure  1  is  the  best  time-to-field  versus  capability  point. 

Similar  to  a  time-to-market  requirement,  the  unified  command’s  assessment  of  when 
a  capability  must  be  fielded  becomes  a  crucial  factor  in  evaluating  the  amount  of  time  to 
devote  to  fielding  a  new  weapons  program  (see  Figure  4).  Timing  plays  a  crucial  role  when 
introducing  a  new  product  and  maximizing  market  share;  similarly,  weapons  must  be  in  the 
hands  of  warfighters  at  the  optimum  moment  on  the  battlefield. 


Market  Pressures  Analogous  To  Meeting  National  Security  Threats 

Time  to  Market  =  Time  to  Field 


Competitive  Environment  Threat 


Figure  4.  Market  Pressures  Analogous  to  Meeting  National  Security  Threats 

Note.  As  the  competitive  environment  for  commercial  goods  and  services  drives  the  necessary  time- 
to-market  to  have  a  market  edge,  the  understanding  and  certainty  of  the  emergence  of  a  threat  to 
national  security  drives  the  required  time-to-field  weapon  systems  and  military  equipment. 
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Just  as  there  is  an  optimum  time  for  a  new  product  to  be  introduced  in  order  to 
capture  the  most  market  share,  there  must  be  some  idea  of  when  a  weapon  must  in  the 
hands  of  the  warfighters  to  achieve  the  desired  effect  on  the  battlefield.  The  time-to-market 
drives  new  product  development  in  the  same  way  that  battlefield  requirements  to  address  a 
threat  drive  weapons  acquisition  programs.  Consequently,  the  time-to-market  demand  will 
be  a  surrogate  for  the  time-to-field  requirement  for  weapon  systems. 

For  the  purpose  of  this  paper,  the  competitive  market  pressures  driving  critical  time- 
to-market  decisions  are  analogues  to  identifying  military  threats  that  drive  the  critical  time-to- 
field.  Competitive  pressures  in  business  have  caused  product  life  cycles  to  compress,  and 
subsequently,  companies  have  taken  measures  to  shorten  their  development  cycles, 
thereby  getting  products  to  market  faster  (Griffin,  1993).  Getting  to  the  fight  with  the  right 
equipment  in  time  to  make  a  difference  against  the  threat  is  a  DoD  priority. 

Value  of  TAIV™  Throughout  Acquisition  Program 

When  TAIV™  is  applied,  there  are  several  benefits  that  result.  The  obvious 
advantage  is  that  an  effective  capability  is  fielded  sooner  and  at  less  cost  to  the  government. 
The  capability  may  be  minimally  sufficient,  but  it  is  sufficient.  Additionally,  TAIV™  has  the 
potential  to  modify  behaviors  that  have  the  potential  to  be  costly,  disrupt  schedule,  and 
threaten  system  performance.  Figure  5  lists  some  of  the  behaviors  that  TAIV™  can 
influence  positively. 

When  the  duration  of  an  acquisition  program  is  constrained  by  specific  timeframes 
with  the  threat  of  cancellation  for  exceeding  those  limits,  events  like  milestones  and  reviews 
will  be  viewed  with  greater  significance.  Developmental  engineering,  systems  engineering, 
and  sustaining  engineering  will  approach  engineering  tasks  constrained  by  time  as 
engineering  challenges  to  be  met  as  they  would  any  other  engineering  tasks.  To  establish 
the  importance  of  TAIV™  as  a  legitimate  and  critical  program  management  tool,  DoD 
acquisition  management  leadership  should  conclude  that  mandating  a  policy  that  threatens 
program  cancellation  for  missing  time-defined  milestones  is  necessary  for  establishing  and 
maintaining  program  internal  control. 


TAIV’*'  Can  Improve  Results  Tied  To  Acquisition  Management  Behaviors 

•  Unambiguous  timeframe  defines  Contractor  and  DoO  management 
performance 

•  Time  constant  that  bounds  the  engineering  tasks 

•  TAIV'"  policy  to  mandate  time-defined  milestones  or  risk  program 
cancellation 

•  Reduces  level  of  optimism  within  DoO  acquisition  community 

•  Reduces  prime  contractor's  inclination  to  over-promise 

•  Helps  to  limit  the  conspiracy  of  hope 

•  Test  community  more  likely  test  to  immediate  requirements 

•  Engineers  less  likely  to  over-design 

•  Time  constrains  limit  the  cost  impacts  of  external  program  influence 

Figure  5.  TAIV™  Can  Improve  Results  Tied  to  Acquisition  Management  Behaviors 

Note.  When  acquisition  program  duration  has  no  time  constraints,  unintended  and  negative  program 
management  behaviors  result.  TAIV™  helps  to  establish  constraints  of  undesired  behaviors. 

Programs  so  often  fall  prey  to  a  misplaced  sense  of  optimism  by  program  managers 
prone  to  believe  that  success  is  inevitable,  despite  evidence  to  the  contrary.  The  constant 
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imposition  of  time  constraints  reduces  that  inclination  toward  optimism.  An  equally  insidious 
and  destructive  behavior  that  grows  with  the  government’s  optimism  is  the  contractor’s 
desire  to  please  the  customer,  which  leads  to  over-promising.  The  result,  according  to 
Kadish  (2006),  was  that  “the  culture  of  the  Department  [of  Defense]  is  to  strive  initially  for  the 
100  percent  solution  in  the  first  article  delivered  to  the  field.”  Kadish  (2006)  went  on  to  say, 
“Further,  the  ‘Conspiracy  of  Hope’  causes  the  Department  to  consistently  underestimate 
what  it  would  cost  to  get  the  100  percent  solution.  Therefore,  products  take  tens  of  years  to 
deliver  and  cost  far  more  than  originally  estimated.” 

The  test  and  evaluation  community  must  be  disciplined  by  the  same  time  constraints 
as  the  rest  of  the  program  management.  That  means  the  Test  and  Evaluation  Master  Plan  is 
constrained  and  managed  to  operate  within  the  program  time  limits.  With  a  mature 
technology  necessary  for  TAIV™  to  be  appropriate,  the  challenges  of  the  test  community 
will  be  correspondingly  reduced  to  live  within  the  TAIV™-established  timeframe. 

Time  constraints  will  by  design  lead  to  shorter  time  cycles,  and  with  those  shorter 
time  cycles  the  tendency  to  “future-proof”  the  weapon  system  or  equipment.  Consequently, 
over-designing  will  be  discouraged,  and  the  temptation  to  try  and  anticipate  a  future  design 
requirement  will  be  less  likely. 

Lastly,  one  of  the  most  disruptive  and  intrusive  influences  comes  from  external 
sources.  Whether  it  is  Congress,  agencies  in  the  executive  branch,  or  the  grass  roots 
activities  of  other  suppliers,  with  limited  time  for  program  execution  comes  limited  time  for 
these  external  actors  to  influence  the  program  execution,  causing  delays,  increased  cost, 
and  potential  reduced  program  performance. 

There  is  a  value  to  TAIV™  that  is  not  immediately  apparent;  but  when  evaluating  the 
challenges  of  developing  an  acquisition  strategy  that  drives  desired  contractor  behaviors,  it 
should  be  considered.  As  part  of  an  in-depth  look  at  “Evaluating  MDAP  [Major  Defense 
Acquisition  Program]  Contractor  Incentives,”  a  research  study  conducted  by  the  NDBI  for  the 
Director,  Performance  Assessment,  and  Root  Cause  Analysis  Directorate,  the  NDBI  found 
that  for  contractor  incentives  to  be  effective,  they  had  to  be  “focused,  clear  and  specific, 
measurable  and  achievable  as  well  as  motivating.”  An  analysis  of  TAIV™  in  this  context  is 
revealing. 

Figure  6  represents  a  comparison  between  total  acquisition  program  cost  on  the  y 
axis  and  program  duration  (time)  on  the  x  axis.  As  time  or  program  duration  increases,  so 
does  the  total  cost  of  the  program.  From  Figure  2,  the  goal  or  objective  time  is  plotted 
against  the  target  delivery  time  and  the  threshold  time  to  field.  In  this  case,  the  threshold 
becomes  the  longest  program  duration  after  which  the  rise  in  cost  (maximum  cost)  makes 
the  program  of  “questionable  value  to  continue.”  Ideally,  the  target  cost  (best  value)  and 
target  delivery  or  fielding  date  are  synchronized,  once  TAIV™  is  exercised. 
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Figure  6.  TAIV™  and  Incentives  Affecting  Contractor  Behavior 

(adapted  from  2004  Monitor  Group  briefing) 

Note.  TAIV™  can  be  used  in  trade  studies  that  reveal  where  to  offer  incentives  to  prompt  contractor 
behavior  that  increases  value  to  the  government.  Larger  incentives  could  be  considered  to  move 
program  delivery  into  the  “Additional  Delivery”  range  for  early  delivery. 

The  contractor  would  be  incentivized  to  strive  to  perform  in  the  green  shaded  area. 
This  is  the  area  where  the  contractor  can  work  to  lower  program  cost  by  delivering  a  quality 
product  earlier.  The  lowering  of  the  program  cost  is  achieved  by  reducing  the  time  to  deliver 
the  weapon  system  to  the  field. 

Larger  incentives  can  be  achieved  by  reducing  the  time  to  deliver  into  the  “additional 
incentive  range.”  Using  TAIV™  creates  the  trade  space  to  fulfill  the  criteria  for  an  incentive 
to  be  effective.  The  incentive  is  focused  on  the  performance  parameter  of  fielding  the 
weapon  system  sooner  than  the  target  delivery  time  and  at  lower  cost  as  a  result. 

The  incentive  is  clear  and  specific  since  establishing  a  time-defined  target  delivery 
point  in  the  duration  of  the  program  execution.  The  contractor  meets  the  target  delivery 
point,  achieves  an  earlier  delivery,  or  misses  the  delivery  target.  Contract  terms  and 
conditions  can  be  crafted  to  provide  a  penalty  for  missing  the  target  delivery  as  a 
disincentive. 

TAIV™  allows  for  the  criteria  for  incentives  to  be  measurable  and  achievable 
because,  again,  the  program  deliverables  either  are  early,  meet  the  target  date,  or  are  late. 
The  analysis  that  accompanies  the  TAIV™  analysis  will  take  into  consideration  the  level  of 
technology  maturity  in  determining  the  target  delivery. 

The  incentive  must  be  motivating.  If  the  previous  three  criteria  are  met,  creating  an 
incentive  with  a  magnitude  of  value  to  the  contractor  becomes  a  matter  of  negotiation.  The 
terms  and  conditions  of  the  contract  establish  the  value  of  the  incentive,  but  with  TAIV™ 
there  is  much  less  ambiguity  around  establishing  the  incentive  value. 

Conclusion 

With  the  emphasis  on  greater  efficiency  in  acquiring  weapon  systems  and 
equipment,  better  buying  power  and  the  more  timely  fielding  of  weapons  can  benefit  from  a 
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more  disciplined  and  structured  approach  to  the  process.  Eliminating  unproductive 
processes  and  bureaucracy  by  reducing  “cycle  times  while  ensuring  sound  investment 
decisions”  (Kendall,  2012)  would  benefit  from  implanting  TAIV™  in  some  form  or  fashion. 

Budget  pressures  are  more  intense  now  than  they  have  been  in  nearly  five  decades. 
Inefficient  use  of  scarce  resources  simply  cannot  be  normal  order.  TAIV™  provides  a 
disciplined  and  structured  process  for  achieving  the  most  capability  and  best-value  cost  in 
the  least  amount  of  time  to  be  effective.  TAIV™  eschews  the  one-size-fits-all  downside  of 
other  approaches  to  reconcile  threat,  time-to-field,  and  cost.  Rather,  TAIV™  is  self-tailoring 
to  prompt  an  appreciation  of  how  using  time  to  establish  boundaries  can  drive  efficient 
acquisition  program  execution. 

The  historical  record  of  acquisition  reform  reports  and  recommendations  is  almost 
unanimous  in  its  view  that,  as  Kadish  (2006)  put  it,  “The  acquisition  process  is  slow,  overly 
complex  and  incompatible  with  meeting  the  needs  of  multiple,  competing,  departmental 
demands,  in  a  diverse  marketplace.”  TAIV™  has  the  potential  to  expedite  the  process  to 
field  the  best  value  defense  product  with  a  level  of  complexity  consistent  with  the 
requirement  to  meet  the  understood  threat.  The  resulting  fielded  capability  will  meet  the 
demands  of  the  only  customer  of  importance — the  warfighter. 
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Introduction 

The  nation’s  military  strategy,  in  large  part,  continues  to  depend  on  superior 
technology,  highly  qualified  operational  forces,  and  the  ability  to  sustain  those  forces  in 
order  to  achieve  its  objectives.  However,  the  global  industrial  base  (as  well  as  the  U.S. 
industrial  base)  no  longer  exists  as  it  did  during  the  Cold  War,  and  the  DoD  must  seek  to 
gain  the  benefits  of  globalization. 

In  the  past,  the  U.S.  industrial  base  would  ramp  up  to  meet  the  needs  of  the  U.S. 
military  and  then  fade  into  the  background  when  the  conflict  was  ended.  Throughout  the 
Cold  War  however,  the  defense  industry  became  a  permanent  segment  of  the  industrial 
base,  providing  dedicated  development  and  production  of  the  systems,  equipment,  and 
supplies.  The  approach  was  to  not  to  mobilize  for  conflict  but  to  have  enough  permanent 


1  This  is  a  summary  of  the  full  report,  which  will  be  available  in  July  2013. 
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capacity  within  the  defense  industry  to  address  it  (Gansler,  1980).  The  industrial  base, 
however,  no  longer  exists  as  it  did  during  the  Cold  War. 

The  Cold  War’s  end  ushered  in  the  following  developments  that  came  to  dominate 
the  restructuring  of  the  defense  industry.  First,  deep  cuts  in  defense  spending  forced  a 
major  consolidation,  down  to  a  small  number  of  defense-dedicated  firms.  Shrinking  defense 
budgets  in  the  1990s  resulted  in  a  string  of  mergers  of  defense  industry  suppliers.  In  1993, 
there  were  21  companies  doing  major  defense  and  aerospace  work;  today,  there  are  six 
U.S. -based  companies:  Boeing,  Lockheed  Martin,  BAE  Systems,  Raytheon,  General 
Dynamics,  and  Northrop  Grumman.  Small  and  large  suppliers  alike — especially  if  they  can 
survive  on  commercial  business  alone — consider  government  accounting  and  reporting 
requirements  burdensome,  and  many  have  stopped  bidding  on  government  contracts, 
thereby  reducing  the  stream  of  suppliers.  In  many  critical  defense  areas,  the  number  of 
suppliers  remaining — at  either  the  prime  contractor  or  lower-tier  levels — is  down  to  only  one 
or  two.  And  with  the  likely  future  stabilization  or  decline  of  defense  budgets,  this 
consolidation  trend  is  potentially  going  to  increase.  Second,  the  commercial  sector  began  to 
invest  heavily  in  high-tech  research  and  development  and  technological  advancement. 

Third,  globally  dispersed  technology  development  and  production  has  left  the  U.S. 
dependent  upon  off-shore  sources  for  critical  defense-related  technologies  (especially  in 
critical,  lower  tier  component  areas).  Finally,  there  was  a  shift  in  emphasis  within  the  DoD 
from  weapons  and  systems  to  complex  communications  and  information  technology.  As  a 
result  of  these,  the  former  U.S.  defense  industry,2  almost  without  exception,  is  transforming 
itself  (through  consolidations,  mergers,  acquisitions,  joint  ventures,  and  integration  that 
crosses  national  boundaries)  into  a  global,  more  commercially  oriented  industry  (Defense 
Science  Board,  1999). 

As  a  result  of  these  four  changes,  the  formally  segregated  defense  industries  of 
Western  countries  are  in  the  process  of  transforming  themselves  (through  consolidations, 
mergers,  acquisitions,  joint  ventures,  and  integrations  that  cross  national  boundaries)  into  a 
global,  more  commercially  oriented  industry.  Take,  for  example,  the  DoD’s  new  MRAP 
vehicles.  They  use  a  V-shaped  hull  that  was  originally  developed  and  refined  in  South 
Africa,  armor  designed  and  developed  in  Israel,  robust  axles  from  Europe,  and  electronics 
from  Asia  (Gansler,  2009).  The  rest  of  the  world’s  defense  industry  is  also  becoming  more 
flat.  Just  recently,  the  United  Arab  Emirates  introduced  a  new  corvette  class  ship,  built  by 
Abu  Dhabi  Shipbuilding. 

But  little  of  what  the  company  featured  came  from  the  UAE.  The  design  of  the 
planned  fleet  of  six  Baynunah-class  ships  originated  at  Constructions 
Mecaniques  de  Normandie  of  Cherbourg,  France.  The  fire  control,  and 
command  and  control  for  the  weapon  systems  came  from  Italy.  The  Exocet 
and  SeaSparrow  missiles  were  built  in  France  and  the  United  States, 
respectively.  South  Africa’s  SAAB  Avitronics  supplied  the  laser  warning 
system.  German  companies  provided  the  decoy  system,  the  sonar,  the 
underwater  communications  and  the  engines.  (Magnuson,  2011) 

This  tendency  toward  globalization — the  tendency  of  markets  for  goods,  services, 
and  capital  to  transcend  national  boundaries  and  become  interconnected — is  not  new;  Ford 
and  General  Motors  were  assembling  cars  in  24  countries  in  1928  (Sturgeon  &  Florida, 
2000).  The  term  globalization  was  first  used  and  identified  as  an  unstoppable  process 


2  The  Europe  Community  has  similar  concerns,  and  is  working  on  developing  a  Community  security 
strategy  and  industrial  policy  (Hartley,  2006;  Markusen  &  Costigan,  1999) . 
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almost  25  years  ago  (Levitt,  1983);  it  has  significantly  accelerated  with  advances  in 
communications  and  computer  technology. 

Current  U.S  defense  trade  and  industrial  policy  does  not  clearly  address 
globalization  or  its  implications.  Instead,  the  current  U.S.  policy  is  the  consolidation  of 
numerous  incremental  changes,  often  contradictory  in  their  aims.  For  example,  the  National 
Security  Strategy  seeks  to  open  markets  and  increase  military  cooperation,  while  export 
controls  and  “buy  American”  laws  inhibit  the  international  trade  in  defense  products 
(McLean,  2005).  Furthermore,  other  factors  such  as  International  Traffic  in  Arms 
Regulations  (ITAR)  and  export  control  laws  disincentivize  commercial  firms  from  entering 
the  defense  market.  When  commercial  technology  has  military  applications,  the  State 
Department  requires  compliance  with  export  control  laws  prior  to  exportation.  These 
restrictions  often  make  commercial  firms  think  twice  before  entering  the  defense  market, 
because  their  goods  may  be  restricted  in  the  commercial  market.  For  example,  in  the 
construction  of  Boeing’s  new  787  Dreamliner,  significant  concern  was  raised  over  similar 
components  that  were  also  used  in  the  Air  Force’s  B-2  Bomber  (Gates,  2006).  Finally, 
restrictions  are  not  made  for  goods  alone  but  can  have  an  impact  on  the  availability  of  labor 
as  well.  For  example,  restrictions  on  security  clearances  or  visas  for  foreign  nationals  often 
make  it  difficult  for  U.S.  firms  to  gain  access  to  the  best  and  brightest  minds  from  around  the 
world  to  work  on  highly  technical  fundamental  research  programs. 

However,  since  the  globalization  of  the  defense  industry  is  a  relatively  recent 
phenomenon,  its  impacts  have  not  yet  been  fully  realized,  or  understood.  Recent 
comprehensive  studies  of  the  U.S.  defense  industry  (since  the  end  of  the  Cold  War  but 
before  the  September  1 1 , 2001 ,  terrorist  attacks)  have  focused  on  the  then  perceived 
overcapacity,  downsizing,  and  conversion  of  the  defense  industry  (Gholz  &  Sopolsky,  1999- 
2000).  Although  acknowledged  as  a  growing  trend,  globalization  is  recognized  for  its 
benefits  along  with  its  risks,  as  well  as  the  lack  of  a  consistent  and  cohesive  national  policy 
(Gansler,  201 1 ;  Markusen  &  Costigan,  1999).  RAND  examined  the  impact  of  globalization 
on  the  defense  aerospace  industry  and  identified  many  benefits,  and  some  risks  but  called 
for  more  research  on  the  issue  (Lorell  et  al.,  2002).  Finally,  a  study  by  the  National 
Research  Council  (Dr.  Jacques  Gansler  participated  in  the  study)  examined  the  availability 
of  the  U.S.  critical  technology  in  a  globalized  environment  and  recommended  the 
development  of  monitoring  capability  of  both  U.S.  industrial  health  and  component 
unavailability  (National  Research  Council,  2004).  Lacking  from  these  studies  is  a 
comprehensive  examination  of  globalization’s  impact  on  the  defense  industrial  base  and 
national  security. 

The  commercial  sector  (which  pays  little  attention  to  national  boundaries)  is  now 
driving  the  development  of  advanced  information  technology,  required  for  most  military 
systems,  and  is  already  very  global.  Manufacturing  industries  were  found  to  be  more 
globalized  in  major  industrial  countries,  although  lagging  in  the  U.S.  (Makhija,  Kim,  & 
Williamson,  1997).  Original  equipment  manufacturers  are  increasingly  contracting  out  their 
manufacturing  and  focusing  exclusively  on  the  product  design  and  marketing  (Sturgeon  & 
Florida,  2000).  Moreover,  the  source  of  competitive  pressure  is  shifting  from  the 
globalization  of  markets  to  the  globalization  of  production,  and  with  it  the  key  competitive 
advantage  has  begun  to  shift  from  excellence  at  the  point  of  production  to  excellence  in 
governing  the  spatially  dispersed  networks  of  plants,  affiliates,  and  suppliers  (Sturgeon  & 
Florida,  2000). 

When  globalization  is  viewed  through  the  prism  of  international  trade,  White  House 
staff  have  argued  that  the  impact  of  increased  job  exportation,  offshoring  (Mankiw  referred 
to  it  as  outsourcing)  is  just  another  form  of  trade  and  “would  ultimately  benefit  the  United 
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States”  (Andrews,  2004).  Other  research  has  supported  this  view  and  concludes  that 
offshoring  leads  to  gains  from  trade  and  increases  to  national  income,  with  minimal  negative 
job  impact  (Bhagwati,  Panagariya,  &  Srinivasan,  2004).  Gomory  and  Baumol  (2000)  argued 
that  the  modern  free-trade  world  is  very  different  from  original  free-trade  models,  and  that 
with  modern  industries,  dominance  can  occur  as  a  result  of  “the  vagaries  of  historical 
accident.”  Once  these  patterns  are  established,  they  tend  to  be  preserved  and  are  less 
influenced  by  free-market  forces;  therefore,  they  suggest  that  government  policy  should 
favor  the  high-value  retainable  industries  (Gomory  &  Baumol,  2000). 

Although  the  globalization  of  the  defense  industry  is  not  a  relatively  recent 
phenomenon,  its  impacts  have  not  yet  been  fully  realized  or  understood. 
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Globalization  Defined 

*  Globalization  is  the  long-term,  largely 
irreversible  phenomenon  involving  the 
political,  cultural,  and  economic  merging  of 
geographically  dispersed  groups  of  people 
across  geopolitical  lines. 

*  Globalization  as  a  concept  has  existed  for 
centuries,  but  only  with  the  advent  of  modem 
transportation  and  communication 
technologies  has  its  application  become  so 
pervasive  and  consequential. 
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“God  did  not  bestow  all  products  upon  all  parts 
of  the  earth,  but  distributed  His  gifts  over 
different  regions,  to  the  end  that  men  might 
cultivate  a  social  relationship  because  one 
would  have  need  of  the  help  of  another.  And  so 
He  called  commerce  into  being,  that  all  men 
might  be  able  to  have  common  enjoyment  of  the 
fruits  of  the  earth,  no  matter  where  produced.” 

-Libanius  (AD  3 14-393).  Orations  (III) 


Today  s Environment _ 

■►Declining  Resources  (with  great  “uncertainty'”) 

*  Rising  Costs  (labor,  equipment,  energy,  health,  etc.) 
■►Demographics  and  debt  payments  adverse  to  needs 
■►Rapidly  Changing  World  (technology’,  economics. 

geopolitics,  etc.) 

*  Globalization  a  reality’  (industry’,  technology, 
economics,  labor,  and  security) 

*  Broad  spectrum  of  security'  concerns  (pirates, 
terrorists,  cvber.  chemical  bio.  nuclear  proliferation, 
“road-side  bombs.  ”  regional  instabilities,  etc.)  -  -  with 
great  "uncertainty’ "(both  in  scenarios  and  in  funding) 
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^8?  The  Message _ 

•  In  General:  Today,  industry,  technology,  and  labor  are 
Globalized  -  -  but,  U.S.  defense  industrial-based  policies  are 

not! 

•  All  Future  Securin'  Scenarios  are  likely  to  be  multi -nation: 
requiring  combined-force  interoperability  -  -  but  U  .S.  export 
controls  largely  limit  this. 


•  hi  many  areas  today,  the  U.S.  is  no  longer  the  technological 
leader  -  -  but  “buy  America”  and  other  inport  controls,  limit 
our  acquisitions;  yet  our  National  Securin'  strategy  is 
“technological  superiority.” 

U.S.  Defense  Industrial  Strategy  Policy  Must 
Change:  In  Order  to  Gain  the  Economic  and 
Security  Benefits  of  Globalization. 


Source:  Center  for  Strategic  and  International  Studies  (CSIS). 
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**  Allies  also  Resource  Constrained _ 

*  "The  economic  crisis  has  hit  our  defence  spending 
hard, "  said  NATO  Secretary  General  Anders  Fogh 
Rasmussen,  addressing  the  NATO  Parliamentary 
A  ssemblv  in  Prague .  " Compared  to  2009,  total 
Allied  defence  expenditure  last  year  declined  by  over 
56  billion  US  dollars  in  real  terns  "  (NATO  Press 
Release,  November  2012). 

*  In  an  era  of  decreasing  defense  budgets,  the  United 
States  and  other  NATO  allies  must  understand  how 
best  to  allocate  their  resources  within  a  global 
market. 


NATO  has  proposed  integrated  Smart  Buying” 
as  the  best  Multinational  approach. 

— nm  in  as  a 


Internationalizing  the  Defense  Industrial  Base 

*The  defense  industrial  base  of  the  United  States  has 
undergone  a  sea  change  in  its  composition,  becoming 
increasingly  reliant  on  international  sources  for  its 
development,  production,  and  provision. 

*  Major  players  in  the  U.  S.  defense  industrial  base  are 
no  longer  solely  domestic. 

■*In  2012.  20  Aerospace  and  Defense  firms  made  the 
Forbes  Global  2000  List  of  the  lamest  public 
companies  operating  in  die  global  market. 
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Market  Value  of  the  Largest  Public  Aerospace  and 

Defense  C  ompany 

Market  Value  of  the  Largest  Global  Public  Aerospace  and  Defense 
Companies 


**>• 


1  caree  fc^n.  » 5* mr.  WAcCorepaes®.  *  Apr!  !f.  2012. 
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impacts  of  Giobalizatiop 

*  Communication 

—  The  advent  of  nev.-  technologies  alleys  for  the  relatively  lav-cost  and,  in 
mam' cases,  instantaneous,  tans  fer  of  large  amounts  of  information. 

—  Borden  are  porous  and  no  longer  easily  enforced. 


*  Culture  and  Education 

—  A  complicated  global  citizenry  etists  -  -  Localises  say  be  split 

—  Foreign  nationals  are  able  to  travel  to  the  United  S  tates  and  obtain  visas  to 
mode  and  attend  school  -  -  ‘Vain  drain  occurs  as  these  people  return  to 
their  countries  of  origin 

*  Economic 

—  Firms  are  increasingly  multinational  in  orientation. 

—  Economic  alliances  and  treaties  are  created  to  promote  mutually benescial 
international  trade  policies  tor  member  states. 

-  I>ial-use  technology  is  made  available  to  the  commercial  market. 

•  eg.  Globa!  ft>atior.sig  Systes  (GPS),  nigit-vrsicr.  technology. etc 

CoKCmMd  t 

tag  *0 
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•  Security  and  Technoloe  y 

-A  greater  need  for  cooperation  amors  states  exists  to  act  jointly  agains  t  other 
states  and,  increasingly,  agains  t  terrorist  organistions  and  other  non -state 
actors. 

-  The  proliferation  of  weapons  and  military  technology  has  become  easier, 
making  for  alliances  among  '  rogue”  states .  For  example 

'Since  the  1960s,  North  Korea's  [weapons]  sales  have  run  the  gainst,  &om 
conventional  v.eapons,  to  increasingly  sophisticated,  longer -range  missiles,  to 
collaborating  v.-ith  Syria  on  the  construction  of  an  entire  clandes  tine  nuclear 
reactor  v.-ith  no  evident  purpose  accept  to  produce  plutonium  far  nuclear 
v.  eapons  ”  (Rosett  'North  Korea  s  Middle  East  Webs  and  Nuclear  Wares." 
Feb.  13,2013) 

—The  rise  of  cvb  er  warfare 

•  Cvbsrapsa  has  tecoce  the  Sff.  dec  in"  d  noderr.  wufn 

•  China  arid  ran  tave  beer.  cpHcaed  ir.sany  cvber  attacks  against  the  US  aer  cepace 
ar.i  defense  industries 

•  The  President  has  been  peer,  broad  autharav'toissue  pj*-e=ptiv»  cvber  athiss.  f  a 
saeisdeenceda  cvber  threat 

•  3etv«n2014and201iUS  Cvber Car=and(CV3ERCOM)willinoeaseas 
votVf  orce  bvJOCP/. 

i  a  a 


Potential  Benefits  of  Strategy /Policy  Change 

^Security: 

-  U.S.  and  allies  would  both  have  State  of  the  Art  (“best  in 
class”) 

-  U.S.  and  allies’  forces  would  be  interoperable  (exercises 
and  war) 

*  Economic: 

-  Economies  of  scale  (from  greater  volume) 

-  Greater  competition  (for  best  performance  at  lowest  cost) 

These  Can  Be  Realized;  While  Always  Directly 
Addressing  Any  Potential  Security  Risks 


a— yyi  ita 
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The  “Good  News” 


♦  In  spite  of  the  domestic  politics,  and  "barriers’"  created, 
steps  are  being  taken 

-  Hie  NRAP  vehicle  (designed  to  harden  against  roadside 
bombs)  uses  armor  designed  in  Israel,  shock  absorbers  from 
Germany  tires  from  France,  and  some  Asian  electronics 


-  AU  U.S.  weapons  have  some  elements  orsinallyfrom  foreign 
sources  -  -  because  of  their  superior  performance 

-  A  feny  leading.  domestically  located,  U.S.  defense  firms  are 
majonty-foreEn-owned  (e.g.,  B.AE  systems;  Finmeccanica; 
EADS;  Thales;  Plasan;  Serco;  etc.  -  -  all  with  “Special 
Security"  Boards) 

President  Obama  Has  Indicated  a  Willingness 
to  Review  Export/Import  Controls 


Joint  Strike  Fighter;  F-35  Lightning  II Program 

♦  The  prime  contractor  of  the  F-35  program  is  the  .American  firm 
L  ockheed  Martin,  with  .American  and  British  firms,  N orthrup 
Grumman  and  B.AE  Systems,  brought  in  as  principal  partners. 

♦  .Additionally,  eight  nations  besides  the  United  States  are  involved  in 
the  F-35’s  iO-year  System  Development  and  Demonstration  (SDD) 
phase:  the  United  Kingdom,  Italy,  the  N  ether  lands.  Turkey,  C  ana  da. 
Denmark,  Norway  and  .Australia. 

♦  By  partnering  with  the  US  during  System  Design  and  Development 
firms  in  these  coimtriescan  "bid  for  work  on  a  best-value  basis,  and 
participate  in  die  aircraft’  s  developm  ent.” 

♦  Israel  and  Singapore  have  also  agreed  to  join  the  program  as 
Security  Cooperation  Participants. 

(Source:  "F-35,”  Joint  Strike  Fighter,  http:  www.jrf.mil  f3 5/.) 
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A 

The  Critical  Labor  Market 

*  National  Securityrequiresthebestin  STEM(Sdence,  Technology, 

Engineering,  and  Math)  -  -  but  U  S .  students  are  not  selecting  these 
fields 

*  Many  Top  U.S.  Universities  and  U.S.  Industry  Research  Centers 
are  establishing  overseas  operations 

*  More  than  half  of  the  graduate  students  in  many  top  U.S. 
Universities,  in  STEM,  are  foreign  students  -  -  who  we  “encourage” 
to  return  home  after  their  studies  (vs.  obtain  citizenship;  if  they  want 

it) 

*  President  Reagan  decided  they  can  work  on  government- funded, 
fundamental  research  (NSDD-1S9);  but  even  this  has  been 
“discouraged’' 

*♦  The  Executive  and  Legislative  branches  are  considering 
increasing  the  number  of  visas  for  STEM  immigrants  (with 
advanced  degrees) 


m 

Initial  Signs  of  Change 


*  Most  U.S.  and  foreign  defense  firms  are  now 
“globalized”  -  -  and  the  trend  is  grow'ing 

*  A  POD  “Study  on  the  Impact  of  Foreign  Sourcing  of 
Systems”  [OSD,  January  2004]  concluded  "utilizing 
foreign  sources  does  not  impact  long-term  readiness:  nor 
impact  the  economic  viability  of  the  national  technology 
and  industrial  base” 

*  The  U.K.  recently  had  a  Navy  ship  built  in  South  Korea 
(for  “best  value”);  and  the  U.S.  is  competing  its  Littoral 
Combat  Ship  between  a  U.S  firm  and  an  Australian  firm. 

I  he  Iechnological.  Economic  and  Security  Potential  Benefits 

of  Globalization  are  Slowly  Being  Recognized;  Now  the 
Strategies  and  Policies  Must  beAdjusted ! 


v\  fuutvxni  rt*  x 
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Abstract 

The  current  Enterprise  Requirements  and  Acquisition  Model,  a  discrete  event  simulation  of 
the  major  tasks  and  decisions  within  the  DoD  acquisition  system,  identifies  several  what-if 
intervention  strategies  to  improve  program  completion  time.  However,  processes  that 
contribute  to  the  program  acquisition  completion  time  were  not  explicitly  identified.  This 
research  seeks  to  determine  the  acquisition  processes  that  contribute  significantly  to  the  time 
a  program  reaches  Milestone  (MS)  B  and  provide  interventions  to  improve  program 
completion  time.  In  order  to  solve  this  problem,  this  research  uses  critical  path  analysis  to 
determine  the  bottleneck  activities  in  the  Pre-MS  B  processes  using  additional  simulation 
analysis.  Results  show  that  the  systems  engineering  processes  are  the  bottleneck  activities 
in  Pre-MS  B  acquisition  stage.  Furthermore,  this  research  then  examines  the  effect  of  these 
processes  by  varying  the  mean  completion  times  and  having  them  occur  earlier  in  the 
acquisition  process.  Potential  policies  are  formulated  from  the  results  to  further  reduce 
program  acquisition  completion  time. 
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Introduction 

A  large  number  of  Department  of  Defense  (DoD)  projects  are  being  completed 
behind  schedule  and  over-budget  (Schwartz,  2010).  To  support  this  claim,  a  Government 
Accountability  Office  (GAO)  report  released  in  2009  stated  that  for  the  DoD’s  2008  portfolio, 
on  average  a  program  faced  a  22-month  delay  and  exceeded  the  original  budget  (Sullivan  et 
al.,  2009).  Generally,  total  cost  growth  of  44%  has  been  consistent  over  the  past  few 
decades  with  a  recent  assessment  by  RAND  (Arena  et  al.,  2006).  Hence,  potential 
intervention  strategies  and  policies  to  improve  the  acquisition  processes  would  be 
worthwhile.  On  the  other  hand,  since  the  end-to-end  DoD  acquisition  process  is  a  large, 
complex,  socio-technological  system,  it  is  difficult  to  analyze  and  determine  which  processes 
or  factors  affect  performance  metrics  like  time,  cost,  and  resource  utilization.  The  current 
DoD  acquisition  system,  which  is  composed  of  three  separate  and  distinct  processes — the 
Joint  Capabilities  Integration  Development  System  (JCIDS),  the  Planning,  Programming, 
Budgeting  &  Execution  (PPBE)  process,  and  the  formal  acquisition  development  system 
outlined  by  the  DoD  5000  series  of  instructions — does  not  exist  in  a  static  environment.  The 
system  is  constantly  being  adjusted,  either  through  policy  changes  or  statute  (CJCS,  2012; 
Weapon  System  Acquisition  Reform  Act  of  2009;  OUSD[AT&L],  2008).  Hence,  other  viable 
analysis  methodologies  must  be  utilized  to  fully  comprehend  this  complex  system. 

In  2009,  a  discrete  event  simulation  (DES)  model  called  the  Enterprise  Requirements 
and  Acquisition  Model  (ERAM)  was  created  by  Wirthlin  (2009).  This  model  was  created  to 
simulate  the  actual  acquisition  processes  of  the  DoD,  using  the  Air  Force  implementation  of 
acquisition  processes  as  the  basis  of  the  model,  in  order  to  provide  further  insight  and 
understanding  of  the  complex  system’s  behavior.  Furthermore,  ERAM  has  benefited  from 
additional  research  since  the  original  2009  Wirthlin  version  (Leach  &  Searle,  2011; 
Montgomery,  2012).  These  new  versions  have  added  additional  functionality  and  options  for 
model  users  to  manipulate  (Wirthlin,  Houston,  &  Madachy,  2011).  According  to  the  ERAM 
model,  during  the  acquisition  process,  approximately  80%  of  the  time,  a  program  was 
undergoing  parallel  processes  when  it  is  in  the  acquisition  system.  It  was  also  observed  that 
one  of  the  main  portions  of  the  model  during  which  these  parallel  processes  take  place  are 
within  the  Pre-Milestone  B  (Pre-MS  B)  stage.  However,  Wirthlin’s  research  did  not  identify 
the  significant  processes  that  affect  the  total  program  time  for  a  project  to  reach  MS  B. 

Against  this  background,  this  research  addressed  these  limitations  and  issues  by 
additional  simulation  and  statistical  analysis  on  the  ERAM  Arena  version  of  the  model.  The 
end  goal  of  this  research  was  to  determine  the  bottleneck  of  the  Pre-MS  B  processes, 
investigate  interventions  to  alleviate  the  bottleneck,  and  translate  them  into  implementable 
policy  changes.  The  rest  of  this  paper  is  organized  as  follows.  The  Review  of  Literature 
section  provides  an  overview  of  the  current  literature  on  bottleneck  analysis  and  the  ERAM 
model.  The  Simulation  Analysis  Methodology  section  presents  the  simulation  analysis 
methodology  performed,  while  the  Results  and  Discussions  section  shows  the  results  of  the 
analysis.  Finally,  the  Conclusions  section  presents  the  conclusions  of  this  research  as  well 
as  viable  intervention  policies  for  reducing  the  time  a  program  takes  to  reach  MS  B. 

Review  of  Literature 

The  ERAM  Model 

The  ERAM  simulation  model  extends  from  the  generation  of  capability  requirements 
in  the  JCIDS  process  to  MS  C,  the  review  before  the  production  stage  begins.  Additionally, 
the  ERAM  is  abstracted  at  a  very  high  level  (Wirthlin,  2009).  This  high  level  of  abstraction 
allows  overall  system  performance  to  be  more  easily  studied.  For  each  replication,  ERAM 
produces  schedule  time  for  programs  that  reach  MS  C.  Although  cost  is  not  measured,  it 
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was  found  that  cost  overruns  were  closely  related  to  schedule  overruns  (Wirthlin,  2009).  The 
validation  and  verification  of  ERAM  included  hand  modeling,  iterations  of  correction  from 
feedback  of  experts  in  all  three  systems  that  comprise  the  entire  acquisition  system,  and 
comparison  of  schedule  and  budget  information  from  the  DAMIR  and  SMART  databases  to 
distributions  of  the  schedule  time  of  model-generated  data  (Wirthlin,  2009). 

The  original  version  of  ERAM  was  created  in  Arena  Simulation  software;  however,  it 
was  translated  into  an  ExtendSim  version  (ERAM  1 .0)  to  serve  as  a  schedule  and  success 
estimation  tool  of  space  programs  for  the  Concept  Design  Center  of  Aerospace  Corporation 
(Leach  &  Searle,  2011).  Leach  and  Searle  further  modified  the  model  introducing  ERAM  1.1 
to  2.1  by  correcting  discrepancies  between  the  Arena  and  ExtendSim  models,  adding  user- 
controlled  variables,  incorporating  space-acquisition  specific  elements,  and  updating  the 
model  to  include  policy  in  the  newly  released  DoDI  5000.02  document.  Montgomery  (2012) 
continued  developing  the  model  in  order  to  add  the  rapid  acquisition  process  and  include 
ACAT  ll/lll  programs.  A  summary  of  the  versions  of  the  ERAM  is  presented  in  Table  1 . 


Table  1.  ERAM  Versions  Adapted  From  Houston  (2012) 


Author 

Version 

Number 

Changes 

Wirthlin  (2009) 

ERAM  1.0 

Baseline  Translation  from  Arena  to  ExtendSim 

ERAM  1.1 

Updates  by  the  Aerospace  Design  Team  and 

Served  as  new  baseline  model 

Leach  and  Searle 

ERAM  1.2 

Implemented  new  DoD  5000.02  policies 

(2011) 

ERAM  2.0 

Incorporated  the  global  variables  that  modify 
acquisition  capabilities 

ERAM  2.1 

Incorporated  the  JCIDS  review  process 

Montgomery  (2012) 

ERAM  2.2 

Added  more  capabilities  for  ACAT  2/3  and  Rapid 
Acquisition  Process 

Since  the  ExtendSim  version  of  ERAM  was  designed  with  the  purpose  of  allowing 
Aerospace  Corporation  to  create  estimates  of  the  schedule  and  success  of  a  particular 
project,  it  has  a  distinctly  different  scope  and  utility  from  the  Arena  model  of  ERAM.  The 
Arena  model  allows  the  user  to  view  the  behavior  of  the  overall  portfolio  while  the 
ExtendSim  version  allows  the  user  to  investigate  a  specific  program.  For  example,  while  the 
ExtendSim  requires  the  user  to  select  a  specific  ACAT  level  for  the  program  being  tested, 
the  Arena  version  assigns  ACAT  levels  based  on  the  distribution  of  programs  observed  in 
the  actual  acquisition  system.  While  the  ExtendSim  version  of  ERAM  was  designed  with  the 
intention  of  allowing  the  user  to  perform  what-if  scenarios,  as  far  as  the  researcher  is 
concerned,  no  literature  of  the  evaluation  of  possible  intervention  strategies  using  the 
ExtendSim  version  of  ERAM  has  been  published.  In  his  dissertation,  Wirthlin  investigated 
the  effect  of  20  interventions  on  the  effect  of  end-to-end  acquisition  time  in  the  Arena 
version.  When  all  20  interventions  were  implemented,  a  20%  reduction  in  end-to-end 
acquisition  time  was  achieved.  However,  more  interventions  can  be  developed  to  further 
study  and  improve  the  DoD  end-to-end  acquisition  process. 

Critical  Path  Analysis 

To  the  best  of  our  knowledge,  no  literature  has  attempted  to  identify  the  critical  path 
of  the  acquisition  process  (Monaco  &  White,  2005).  Although  long  cycle  times  continue  to 
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plague  DoD  acquisition  programs,  relatively  few  studies  have  focused  on  identifying 
significant  processes  that  dictate  program  cycle  time.  Despite  the  Packard  Commission’s 
assertion  that  schedule  drives  costs,  most  studies  and  policy  changes  have  focused  on  cost 
reduction  rather  than  reducing  cycle  time  (Al-Harbi,  2001;  McNutt,  1999).  Drezner  and  Smith 
(1990)  performed  a  statistical  analysis  of  10  programs  in  order  to  hypothesize  factors  that 
affect  the  original  plan  and  program  deviation.  A  study  performed  by  Tyson,  Nelson,  Om, 
and  Palmer  (1989)  examined  schedule  variance  and  its  causes.  The  study  found  that 
prototyping,  sole-source  procurement,  fixed-priced  contracts,  and  multiyear  procurement 
reduced  schedule  variance.  The  study  also  found  that  programs  awarded  through  full  and 
open  competition  experience  more  schedule  growth  than  those  programs  that  did  not. 
Another  possible  schedule  driver  is  presented  by  Brown,  Flowe,  and  Hamel  (2007).  Brown 
et  al.  compared  the  schedule  quality  of  joint  and  single-system  programs.  From  this  study  it 
was  found  that  joint  system  programs  have  significantly  more  schedule  breaches;  however, 
the  research  did  not  identify  the  root  cause  of  this  difference  (Brown  et  al.,  2007). 

In  summary,  to  the  best  of  our  knowledge,  there  exists  no  research  that  has  been 
conducted  that  isolates  and  identifies  bottleneck  activities  and  its  effect  on  the  program 
completion  time  throughout  the  DoD  acquisition  process.  Hence,  intervention  strategies  to 
be  developed  must  be  focused  on  addressing  bottleneck  issues  to  obtain  maximum 
improvement  of  the  end-to-end  DoD  acquisition  process. 

Simulation  Analysis  Methodology 

This  section  describes  the  analysis  performed  to  identify  bottleneck  operations  within 
the  Pre-MS  B  stage.  After  identifying  bottleneck  operations,  intervention  strategies  were  also 
formulated  in  this  section  to  reduce  total  program  completion  time.  Hence,  this  research  was 
performed  in  two  phases.  A  brief  description  of  these  phases  is  presented  as  follows: 

•  The  first  phase  performed  a  critical  path  analysis  on  the  Pre-MS  B  activities  to 
identify  a  bottleneck  (see  the  Identification  of  Bottleneck  Activities  subsection). 

•  The  second  phase  focused  on  investigating  the  effect  of  reducing  the  process 
times  of  the  identified  bottleneck  activities  from  Phase  1  and  determining  the 
effect  of  allowing  them  to  be  executed  earlier  in  the  process  (see  the  Design  of 
Pre-MS  B  Bottleneck  Interventions  subsection). 

Identification  of  Bottleneck  Activities 

In  order  to  perform  critical  path  analysis,  the  Pre-MS  B  section  was  mapped  by  hand 
to  assist  in  visualization  of  the  complex  network  of  separation  and  batches  in  the  acquisition 
system.  The  processes  between  each  Separate  and  Batch  method  were  left  out  for 
simplicity  and  ease  of  interpretation.  The  section  or  line  segment  between  any  two  nodes 
was  labeled.  Figure  1  shows  the  mapped  version  of  the  Pre-MS  B  activities  and  Table  2 
shows  the  activities  associated  with  each  section. 
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1 


Figure  1.  Pre-MS  B  Flowchart 


Table  2.  List  of  Activities  in  the  Pre-MS  B  Flowchart 

Section  Description  of  Activities 

1  Requirements  generation:  KPP  Development,  high  performance  team  work, 
etc. 

2  RFP  release,  contract  awarding 

3  Waiting  period  for  start  of  contract 

4-6  Cost  estimates  (contractor,  program  office,  and  independent) 

7  Affordability  assessment 

8  Set  acquisition  program  baseline 

9-10  No  processes 

1 1  Prepare  and  conduct  acquisition  panels 

12  Early  Systems  Engineering  (SE)  activities:  EOA,  developmental  testing, 
SRR,  etc. 

13  Acquisition  planning  activities 

14  Draft  RFP 

15  RFP  coordination 

16  Source  selection  plans 


Several  Assign  and  Record  modules  were  added  to  the  Arena  model  in  order  to 
determine  the  time  to  complete  each  segment.  Next,  a  trial  of  3,000  runs  was  performed  in 
Arena,  and  the  times  for  each  segment  were  collected.  A  spreadsheet  was  then  used  to 
analyze  results  and  to  determine  the  time  of  every  possible  path  from  the  beginning  of  the 
Pre-MS  B  activities  to  the  MS  B  decision.  The  path  that  took  the  longest  amount  of  time  was 
deemed  as  the  critical  path.  By  comparing  segments  in  the  longest  paths  to  the  sections 
found  in  shorter  paths,  the  bottleneck  activities  for  the  Pre-MS  B  processes  were  identified. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-  52- 


Design  of  Pre-MS  B  Bottleneck  Interventions 

In  order  to  improve  the  performance  and  alleviate  the  delay  caused  by  this 
bottleneck,  two  intervention  strategies  were  developed  and  tested  in  ERAM.  The  first 
intervention  performed  was  to  test  the  effect  of  decreasing  the  process  time  for  all 
bottleneck  activities.  In  order  to  test  the  effect  of  reducing  total  process  time,  the  minimum, 
maximum,  and  mode  for  these  activities  was  reduced  by  a  fixed  percentage.  A  paired  t-test 
was  then  performed  to  compare  each  trial  to  the  baseline  at  95%  confidence  level.  The 
reduction  by  using  a  fixed  percentage  was  performed  until  a  statistically  significant  change 
was  obtained.  Furthermore,  the  second  intervention  was  a  sensitivity  analysis  to  determine 
the  effect  of  allowing  the  bottleneck  to  be  performed  earlier  in  the  Pre-MS  B  process  to 
determine  its  effect  on  the  total  process  time.  The  results  of  these  interventions  are 
illustrated  in  the  next  section. 

Results  and  Discussions 

This  section  presents  the  results  of  both  simulation  analysis  phases  performed  on 
the  ERAM  Arena  model.  Specifically,  the  Pre-MS  B  Critical  Path  Analysis  Results 
subsection  presents  the  results  of  the  identification  of  the  critical  path  and  bottleneck 
activities.  Additionally,  the  Additional  Pre-MS  B  Bottleneck  Interventions  subsection  shows 
the  results  of  the  interventions  performed  on  the  bottleneck  analysis  to  improve  program 
completion  time. 

Pre-MS  B  Critical  Path  Analysis  Results 

During  the  critical  path  analysis,  times  for  all  1 1  paths  through  the  system  were 
calculated.  The  paths  were  labeled  by  letters.  Each  path  was  composed  of  segments.  A 
subset  of  the  paths  and  their  corresponding  activities  is  shown  in  Table  3. 


Table  3.  List  of  Paths  and  Segments  for  Pre-MS  B 


Path  Name 

Corresponding  Segments  From  Figure  1 

A 

1 

B 

2,  12,  14,  16,  10,  11 

C 

2,  12,  14,  15,  10,  11 

D 

2,  3,  6,  7,  8,  11 

E 

2,  3,  5,  7,  8,  11 

F 

2,  3,  4,  7,  8,  11 

G 

2,  3,  6,  7,  9,  10,  11 

H 

2,  3,  5,  7,  9,  10,  11 

1 

2,  3,  4,  7,  9,  10,  11 

J 

2,  3,  13,  14,  15,  10,  11 

K 

2,  3,  13,  14,  16,  10,  11 

As  seen  from  the  Table  3,  paths  B  and  C  heavily  overlap  while  path  A  has  no  overlap 
with  any  other  path.  From  the  total  time  for  each  path,  the  longest  was  deemed  the  critical 
path.  The  second  longest  and  third  longest  paths  were  also  determined.  A  subset  of  this 
data  can  be  seen  in  Table  4. 
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Table  4.  Length  of  Longest  Paths  to  MS  B 


Run 

Percentage 
of  Runs  as 
Longest  Path 

Percentage  of 
Runs  as 
Second 
Longest  Path 

Percentage  of 
Runs  as  Third 
Longest  Path 

Percentage  of  Runs 
in  the  Top  Three 
Longest  Path 

A 

45 

8.75 

42.5 

96.25 

B 

43.75 

38.75 

7.5 

90.00 

C 

6.25 

38.75 

31.25 

76.25 

D 

3.75 

8.75 

8.75 

21.25 

F 

0 

2.5 

5 

7.5 

J 

0 

1.25 

1.25 

2.50 

K 

1.25 

1.25 

3.75 

6.25 

E,  G,  H, 1 

0 

0 

0 

0 

As  can  be  observed  in  Table  4,  the  critical  path  was  most  often  A,  B,  and  C.  In 
approximately  95%  of  the  trials,  either  A,  B,  or  C  composed  the  critical  path.  Specifically, 
50%  of  the  time  path  B  or  C  was  the  critical  path,  and  45%  of  the  time  path  A  was  the  critical 
path.  We  note  that  path  B  and  C  have  significant  overlap;  therefore,  they  are  considered  as 
a  single  path,  path  B/C.  Since  the  critical  path  was  very  evenly  split  between  path  A  and 
path  B/C,  it  can  be  deduced  that  a  Pre-MS  B  process  common  to  both  of  paths  would  be  the 


bottleneck  of  the  process. 

In  examining  the  ERAM,  it  can  be  gleaned  that  there  was  some  interaction  between 
path  A  and  path  B/C.  One  of  the  last  modules  of  path  A  was  a  hold  module  called  “Wait  for 
EOA  completion.”  A  screenshot  of  this  module  can  be  seen  in  Figure  2. 


Figure  2.  Wait  for  EOA  Completion  Screenshot 


As  seen  in  Figure  2,  path  A  must  wait  for  the  EOA  to  be  complete  before  the  path 
can  finish.  A  second  communication  occurs  between  the  two  paths.  In  order  for  the  SE 
activities,  like  the  EOA,  to  occur,  the  Key  Performance  Parameters  (KPPs)  must  be 
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complete.  The  hold  model  called  “Wait  for  T&E  start”  facilitates  this  communication  and  can 
be  seen  in  Figure  3. 


Figure  3.  Wait  for  T&E  start  Screenshot 


However,  we  note  that  this  hold  module  also  waits  for  the  75%  of  the  contract  length 
to  elapse.  At  the  default  settings,  the  KPPs  will  always  be  completed  in  less  than  75%  of  the 
contract  length.  Therefore,  at  the  default  settings,  this  hold  does  not  serve  as 
communication  between  the  paths. 


Since  the  completion  of  the  EOA  was  the  only  communication  between  the  two 
critical  paths,  the  SE  activities  that  begin  before  the  EOA  completion  was  determined  to  be 
the  bottleneck  of  the  Pre-MS  B  activities.  If  this  bottleneck  activity  were  removed,  the  time  to 
MS  B  would  be  reduced  by  an  average  of  6.8%. 


Additional  Pre-MS  B  Bottleneck  Interventions 

Table  5  summarizes  the  results  of  the  t-tests  performed  for  when  the  process  time 
for  MS  B  system  engineering  activities  was  reduced.  The  tables  show  a  subset  of  trials 
corresponding  to  a  reduction  in  process  times  by  0%,  20%,  35%,  or  50%.  These  settings 
were  selected  to  show  the  sensitivity  of  the  model  to  various  degrees  of  process  time 
reduction.  From  these  simulation  analyses,  the  mean  ith% )  and  standard  deviation  of  the 
total  completion  time  for  each  trial  were  calculated.  These  calculated  means  were  compared 
to  the  mean  of  the  baseline  setting  (iibaSe)  in  the  default  settings,  or  0%  process  time 
reduction.  The  null  hypothesis  for  the  t-tests  is  H0-.  nbase  =  ii jth%  which  corresponds  to  a 
failure  to  reject  the  claim  that  the  baseline  and  the  ith  percentage  are  similar  and  alternative 
hypothesis  Ht\  iibase  =£  nith0/o  if  there  is  significant  difference. 
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Table  5. 


Summary  of  t-Test  Results  of  Process  Time  Reduction  for  SE  Activities 

%  Reduction  of  Process  Time 


0% 

20% 

35% 

50% 

(Baseline) 

Average  Time  to  MS  B 
(Days) 

3418.01 

3274.90 

3211.564 

3164.25 

Standard  Deviation  (Days) 

1701.08 

1636.108 

1557.816 

1515.48 

P-Value 

0.281 

0.109 

0.046 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Reject 

Ho 

Ho 

Ho 

As  seen  in  Table  5,  it  is  evident  that  when  the  process  time  for  SE  activities  was 
decreased  by  less  than  50%,  there  will  not  be  a  statistically  significant  decrease  in  the  time 
to  MS  B.  However,  when  the  process  times  for  SE  activities  are  reduced  by  more  than 
approximately  50%,  the  model  exhibits  a  statistically  significant  decrease  in  time  to  MS  B. 


Based  on  the  identified  bottleneck,  which  was  the  SE  activities,  a  second  intervention 
was  developed.  Specifically  a  sensitivity  analysis  was  done  to  test  the  effect  of  allowing  the 
bottleneck  activities  to  occur  earlier  in  the  contract.  This  was  implemented  by  adjusting  the 
module  called  “Begin  Testing  PreB,”  which  can  be  seen  in  Figure  4. 


Figure  4.  Begin  Testing  PreB  Screenshot 


The  “Begin  Testing  PreB”  module  is  a  Decide  module  that,  when  set  to  true,  triggers 
the  beginning  of  the  SE  activities.  The  original  criteria  for  the  decide  module,  as  verified  and 
validated  by  Wirthlin  when  creating  the  ERAM  model,  was  that  75%  of  the  contract  length 
must  pass  before  these  activities  can  occur.  During  this  research,  this  percent  was 
decreased  to  simulate  the  SE  activities  occurring  sooner  and  more  resources  being  applied 
at  the  beginning  of  the  contract. 

In  addition,  to  allow  SE  tasks  to  begin  sooner,  the  KPPs  must  be  completed  sooner 
in  the  process  as  their  completion  is  also  needed  to  trigger  the  start  of  the  SE  tasks.  A  more 
complete  discussion  of  this  interaction  can  be  found  in  the  Pre-MS  B  Critical  Path  Analysis 
Results  subsection.  The  process  time  of  the  KPP  development  was  reduced  in  order  for  the 
KPPs  to  be  completed  in  a  manner  that  does  not  delay  the  SE  activities.  A  paired  t-test  was 
then  performed  to  compare  each  trial  to  the  baseline  at  95%  confidence  level. 
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Tables  6  and  Table  7  summarize  the  results  of  the  t-tests  performed  for  allowing  Pre- 
MS  B  contractor  activities  to  occur  earlier  in  the  contract.  Specifically,  Table  6  shows  the 
effect  of  allowing  the  SE  activities  to  occur  earlier  in  the  contract  when  the  KPPs  generation 
process  time  was  not  decreased,  and  Table  7  shows  the  effect  of  allowing  the  SE  activities 
to  occur  earlier  in  the  contract  in  conjunction  with  the  KPPs  generation  process  performing 
faster.  Table  6  shows  a  subset  of  trials  corresponding  to  the  SE  activity  starting  when  75%, 
50%,  33%,  or  25%  of  the  contract  has  elapsed.  Table  7  shows  a  subset  of  trials 
corresponding  to  the  SE  activity  starting  when  75%,  65%,  60%,  or  55%  of  the  contract  has 
elapsed. 

These  settings  were  selected  to  show  the  sensitivity  of  the  model  to  various  start 
times  of  SE  activities.  From  these  simulations,  the  mean  (nith %)  and  standard  deviation  of 
the  total  MS  B  completion  time  for  each  trial  were  calculated.  These  calculated  means  were 
compared  to  the  mean  of  the  baseline  setting  ( nbase )  'n  the  default  settings,  or  starting  after 
75%  of  the  contract  has  elapsed.  The  null  hypothesis  for  the  t-tests  is  H0\ iibase  =  [iith0/o 
which  corresponds  to  a  failure  to  reject  the  claim  that  the  baseline  and  the  ith  percentage  of 
contract  elapsing  before  start  is  similar  in  terms  of  program  completion  time  and  alternative 
hypothesis  //iibase  *  ^  there  is  significant  difference. 


Table  6.  Summary  of  t-Test  Results  of  SE  Activity  Start  Time  Adjustments  With 

Original  KPP’s  Process  Time 


%  of  Contract  Elapsed  Before  Start 

75% 

(Baseline) 

50% 

33% 

25% 

Average  Time  to  MS  B 
(Days) 

3418.01 

3379.09 

3379.09 

3379.09 

Standard  Deviation  (Days) 

1701.08 

1670.31 

1670.31 

1670.31 

P-Value 

0.770 

0.770 

0.770 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Fail  to  Reject 

Ho 

Ho 

Ho 

Table  7.  Summary  of  t-Test  Results  of  SE  Activity  Start  Time  Adjustments  With 

Reduced  KPP’s  Process  Time 


%  of  Contract  Elapsed  Before  Start 

75% 

(Baseline) 

65% 

60% 

55% 

Average  Time  to  MS  B 
(Days) 

3418.01 

3305.44 

3200.75 

3139.95 

Standard  Deviation  (Days) 

1701.08 

1628.08 

1599.04 

1553.38 

P-Value 

0.392 

0.099 

0.032 

Conclusion 

Fail  to  Reject 

Fail  to  Reject 

Reject 

Ho 

Ho 

Ho 
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As  seen  in  Table  6,  it  is  evident  that  the  time  to  MS  B  is  not  sensitive  to  an  earlier 
start  time  for  SE  activities  when  the  KPP  process  time  is  set  to  the  default  distribution.  In 
fact,  when  the  start  time  is  at  50%,  33%,  and  25%  of  the  contract  time,  the  time  to  MS  B, 
standard  deviation  of  time  to  MS  B,  and  p-value  are  identical.  This  is  due  to  the  hold  module 
in  the  SE  path  described  earlier.  As  previously  discussed,  in  order  for  the  SE  activities  to 
begin,  a  percent  of  the  contract  must  elapse  and  the  KPPs  must  be  complete.  Once  the  SE 
activities  start  time  occurs  earlier  than  50%  of  the  contract  length,  the  KPPs  completion  is 
the  determining  factor  of  the  SE  activity  start  time. 

Table  7  takes  this  into  account  by  reducing  the  KPPs’  process  time  to  a  point  where 
it  does  not  dictate  the  start  of  the  SE  activities.  As  seen  in  Table  7,  it  is  evident  that  when  SE 
activities  begin  at  60%  of  the  contract  length  or  later,  there  will  not  be  a  statistically 
significant  decrease  in  the  time  to  MS  B.  However,  when  SE  activities  begin  at  55%  of  the 
contract  length  or  sooner  and  the  KPPs’  generation  processes  are  shortened  to  the  same 
degree,  the  model  exhibits  a  statistically  significant  decrease  in  time  to  MS  B. 

Conclusions 

The  critical  path  analysis  performed  in  this  research  indicated  that  the  SE  activities 
and  their  communication  with  the  requirements  branch  are  the  bottleneck  of  the  Pre-MS  B 
portion  of  the  acquisition  system.  In  addition,  the  research  indicated  that  focusing  on  reforms 
that  address  this  bottleneck  has  the  potential  to  decrease  the  total  time  spent  on  MS  B 
activities  by  approximately  7%;  this  corresponds  to  a  process  time  reduction  of 
approximately  six  months. 

This  research  also  tested  two  strategies  to  address  this  bottleneck.  The  first  was 
reducing  the  process  time  of  all  SE  activities.  The  second  was  to  allow  the  SE  activities  to 
have  an  earlier  start  time.  This  research  showed  that  the  latter  policy  has  the  potential  to  be 
the  most  beneficial.  This  research  showed  that  the  process  times  for  all  SE  activities  must 
be  decreased  by  approximately  50%  in  order  for  a  statistically  significant  decrease  in  time  to 
MS  B  to  occur.  This  degree  of  process  time  reduction  may  be  infeasible.  On  the  other  hand, 
allowing  the  SE  activities  to  occur  after  55%  of  the  contract  time  has  elapsed  rather  than  the 
current  75%  produces  a  statistically  significant  decrease  in  time  to  MS  B. 

The  increased  sensitivity  of  program  time  to  start  time,  rather  than  process  length, 
suggests  that  schedule  benefits  may  be  achieved  if  the  some  resources,  both  financial  and 
human,  are  transferred  from  the  SE  activities  to  the  activities  prior  to  test  and  development. 
However,  this  re-allocation  of  resources  must  be  accompanied  by  responsiveness  from  the 
JCIDS  branch,  which  is  the  branch  that  generates  the  KPPs.  This  research  indicates  that 
there  was  a  large  amount  of  co-dependence  between  the  JCIDS  and  SE  activities  and  that 
communication  and  coordination  between  these  branches  is  needed  in  order  to  address  the 
bottleneck. 
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Abstract 

In  systems  today,  software  provides  substantial  portions  of  capability  and  performance.  In 
many  Department  of  Defense  (DoD)  acquisitions,  however,  software  too  often  is  a  minor 
consideration  when  the  early  and  most  constraining  decisions  about  program  cost,  schedule, 
and  behavior  are  made  (i.e.,  prior  to  Milestone  A).  These  decisions  manifest  themselves  in 
the  acquisition  strategy,  the  system  and  software  architecture,  and  ultimately  in  the  deployed 
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system.  Based  on  our  experience  with  large  programs,  we  have  identified  seven  patterns  of 
failure  that  lead  to  misalignment  between  the  software  architecture  and  the  acquisition 
strategy,  leading  to  program  restarts,  cancellations,  or  other  failures. 

We  describe  the  characteristics  of  these  patterns  of  failure  and  relate  them  to  weak  or 
missing  relationships  between  key  artifacts — relationships  that  should  exist  even  at  an  early 
stage  of  the  life  cycle.  In  this  paper,  we  focus  on  those  artifacts  that  relate  to  the  expression 
and  analysis  of  business  goals.  We  present  early  results  in  the  development  of  acquisition 
quality  attributes  (analogous  to  software  quality  attributes)  and  how  these  attributes  relate  to 
acquisition  strategies.  We  conclude  with  some  speculation  on  what  is  needed  to  avoid  the 
failure  patterns. 

Introduction 

In  systems  today,  software  provides  substantial  portions  of  capability  and 
performance.  One  consequence  for  DoD  acquisition  is  that  software  issues  are  major  factors 
in  cost  and  schedule  overruns.  However,  software  is  often  a  minor  consideration  when  the 
early,  most  constraining  program  decisions  are  made.  This  has  negative  consequences, 
most  often  in  terms  of  misalignments  between  the  software  architecture2  and  system 
acquisition  strategies.3  Our  analysis  of  troubled  programs  shows  that  these  misalignments 
lead  to  program  restarts,  cancellations,  and  failure  to  meet  important  mission  and  business 
goals.4  Our  research  focuses  on  enabling  organizations  to  reduce  their  program  failures  by 
harmonizing  their  acquisition  strategy  with  their  system  and  software  architectures. 

One  major  source  of  misalignments  is  the  diverse  sets  of  stakeholders  of  complex 
programs;  it  is  inevitable  that  some  (perhaps  many)  of  their  diverse  goals  and  priorities  are 
in  conflict.  Operational  users,  combat  commanders,  funding  authorities,  and  acquisition 
team  members  may  think  they  share  the  same  priorities,  but  when  interviewed,  their 
answers  often  vary  widely  in  terms  of  the  goals  and  features  they  see  as  the  most  important. 
In  many  cases,  the  solutions  that  are  then  created  are  based  on  goals  of  one  set  of 
stakeholders — goals  that  can  conflict  with  those  of  other  stakeholders  without  explicit 
consideration  of  the  tradeoffs  being  made.  Ultimately,  such  conflicts  in  goals  eventuate  in 
the  misalignments  described  previously. 

Observed  Patterns  of  Failure 

Characteristics  of  the  Patterns  We  Observed 

The  major  source  of  our  data  for  this  research  was  interviews  with  participants  in 
major  acquisition  programs,  most  of  which  encountered  some  sort  of  difficulty,  ranging  from 
partial  to  total  failure.  These  data  are  also  supported  by  decades  of  experience  on  the  part 
of  all  of  the  authors  in  studying,  assessing,  and  participating  in  actual  programs,  many  of 
which  evidenced  similar  failing  behaviors.  Virtually  all  of  the  conclusions  derived  from  our 


2  Software  architecture  is  defined  by  Bass,  Clements,  and  Kazman  (2012)  as  “the  structure  or 
structures  of  the  system,  which  comprise  software  elements,  the  externally  visible  properties  of  those 
elements,  and  the  relationships  among  them.” 

3  Acquisition  strategy  (2011)  is  defined  by  the  Defense  Acquisition  University  as  “a  business  and 
technical  management  approach  designed  to  achieve  program  objectives  within  the  resource 
constraints  imposed.  It  is  the  framework  for  planning,  directing,  contracting  for,  and  managing  a 
program.  It  provides  a  master  schedule  for  research,  development,  test,  production,  fielding, 
modification,  postproduction  management,  and  other  activities  essential  for  program  success.” 

4  A  mission  goal  is  an  expression  of  an  objective  that  affects  a  user,  focused  on  what  the  solution  or 
product  should  do  or  how  it  should  behave.  A  business  goal  is  an  expression  of  an  organizational 
(e.g.,  Navy)  objective,  focused  on  goals  relative  to  the  business  model  for  acquiring  or  developing  the 
solution  or  product. 
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interviews  were  strongly  supported  by  our  aggregate  experiences  as  active  professionals,  in 
both  government  and  non-government  roles  in  the  domain  of  DoD  acquisition.  Analysis  of 
these  data  has  led  us  to  observe  several  recurring  patterns.  Because  of  their  negative 
consequences,  and  following  common  usage  throughout  the  software  engineering 
community  as  exemplified  by  Brown,  Malveau,  McCormick,  and  Mowbray  (1998),  we 
characterize  these  as  anti-patterns. 

Buschmann,  Meunier,  Rohnert,  Sommerlad,  and  Stal  (1996)  provided  a  form  for 
describing  patterns,  calling  out  the  need  to  give  patterns  a  name  and  to  define  the  context 
(environment),  the  problem,  and  the  solution.  We  modify  this  approach  for  our  anti-pattern 
descriptions.  First,  we  include  the  notion  of  consequence,  as  defined  by  Gamma,  Helm, 
Johnson,  and  Vlissides  (1994).  Second,  in  lieu  of  calling  the  observed  activity  “solution,”  we 
have  titled  that  activity  “observed  response”  (i.e.,  to  the  problem).  Thus,  each  of  our  anti¬ 
pattern  descriptions  has  the  following  elements: 

•  name  (readily  identifies  some  key  aspect  of  the  problem) 

•  context  (situations  where  the  pattern  occurs) 

•  problem  (recurring  issues  and  forces  that  characterize  the  problem) 

•  observed  response  (the  manner  in  which  people  attempt,  consciously  or 
otherwise,  to  solve  the  problem) 

•  consequences  (results  of  applying  the  observed  response  to  the  problem  in 
the  given  context) 

Overview  of  Findings 

In  each  of  the  programs  studied,  we  observed  several  instances  of  activities  and 
behaviors  that  qualify  as  anti-patterns.  In  some  cases,  the  anti-patterns  were  of  sufficient 
magnitude  that  they  had  severe  negative  effects  on  program  success.  None  of  the  anti¬ 
patterns  were  observed  in  all  of  the  programs  we  studied,  and  none  were  seen  in  only  one 
program.  We,  therefore,  believe  that  they  were  sufficiently  pervasive  that  they  were  true 
patterns  of  behavior. 

The  set  of  anti-patterns  we  observed  in  these  programs  consists  of  the  following: 

1.  Undocumented  Business  Goals 

2.  Unresolved  Conflicting  Goals 

3.  Failure  to  Adapt 

4.  Turbulent  Acquisition  Environment 

5.  Poor  Consideration  of  Software 

6.  Inappropriate  Acquisition  Strategies 

7.  Overlooking  Quality  Attributes 

These  anti-patterns,  like  most  factors  that  negatively  affect  acquisition  programs, 
eventually  result  in  a  small  number  of  familiar  and  unhappy  consequences:  schedule  delays, 
cost  overruns,  delivery  of  less  than  was  promised,  or  outright  program  cancellation.  In  that 
sense,  the  ultimate  consequences  of  these  anti-patterns  in  the  programs  we  studied  have 
an  expected  similarity. 

However,  the  immediate  consequences  of  different  anti-patterns  differ  in  many  ways. 
For  example,  the  immediate  effects  of  leaving  key  business  goals  unstated  is  different  from 
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those  that  result  from  a  turbulent  acquisition  environment.  Since,  in  the  methods  we  hope  to 
develop,  we  intend  to  focus  on  some  of  the  immediate  and  visible  symptoms  of  anti-patterns 
as  a  way  to  minimize  or  eliminate  their  negative  influence,  these  immediate  consequences 
are  important.  Therefore,  in  the  descriptions  that  follow,  we  indicate  a  number  of  these 
immediate  consequences,  although  we  also  mention,  wherever  possible,  the  longer  term 
consequences  that  we  observed. 

Each  of  these  is  discussed  in  the  subsections  that  follow. 

Undocumented  Business  Goals 

Context.  This  anti-pattern  stems  from  a  lack  of  precise,  well-defined,  and  well- 
documented  business  goals  for  a  DoD  acquisition,  goals  that  would  correspond  to  the 
precise,  well-defined  mission  goals  usually  created  for  a  program.  The  DoD’s  business  goals 
are  the  major  driver  for  an  acquisition  strategy,  which  should  make  some  provision  for 
software.  But  the  actual  role  that  the  detailed  business  goals  play  in  the  software, 
particularly  its  architecture,  is  often  minimal. 

Problem.  Although  business  goals  obviously  influence  a  program’s  acquisition 
strategy,  they  can  also  have  a  strong  influence  on  system  and  software  architecture.  This 
additional  influence  is  seldom  recognized,  even  when  it  is  vital  that  it  should  be.  Examples  of 
such  influence  include  the  following: 

•  “avoid  vendor  lock”  or  “maximize  competition”  has  potentially  significant 
importance  when  defining  software  architecture; 

•  a  mandate  to  employ  reuse  as  much  as  possible  may  have  a  strong  negative 
impact  on  the  software  architecture  if  the  software  to  be  reused  is  itself  poorly 
architected;  and 

•  the  goal  of  a  software  “open  architecture”  may  have  a  significant  impact  on 
the  underlying  system  architecture.5 

Two  factors  cause  this  problem.  First,  at  the  high  level,  many  business  goals  are 
generally  expressed  only  as  very  broad  mandates,  and  others  are  not  explicitly  expressed  at 
all.  Second,  at  the  detailed  level,  there  is  no  useful  process  for  capturing  and  prioritizing 
business  goals  in  a  program-specific  way  that  is  comparable  to  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  process  that  supports  definition  and 
prioritization  of  mission  goals  and  their  associated  requirements. 

This  anti-pattern  is  particularly  problematic  in  programs  that  are  building  a  system 
that  must  integrate  with  other  systems  (which  may  themselves  be  in  varying  stages  of 
development).  While  the  high-level  goal  of  an  integrated  system  may  be  explicit,  the  detailed 
goals  (together  with  an  understanding  of  needed  resources)  for  the  system  of  systems 
(SoS)  are  often  left  unspecified. 

Observed  Response  to  the  Problem.  The  general  response  to  this  anti-pattern  is 
that  the  architect  has  no  other  choice,  and  hence  the  mission  requirements  defined  by  the 
operational  side  drive  the  architecture,  which  then  reflects  only  those  mission  requirements. 
Yet,  were  the  detailed  business  goals  available,  some,  perhaps  many,  of  those  business 
goals  might  be  critical  enough  to  overshadow  some  of  the  mission  requirements. 


5  Using  Maier  (2006),  we  characterize  typical  system  architectures  as  compositional  (is-a-part-of 
relationships)  and  software  architectures  as  more  typically  layered  hierarchies  (is-used-by 
relationships). 
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An  example  can  be  seen  in  an  instance  where  there  might  be  an  implicit  but 
unspecified  business  goal  that  favored  a  highly  distributed  contractor/subcontractor  profile. 
Lacking  awareness  of  that  goal  (because  it  is  unstated),  a  software  architect  might 
reasonably  design  a  monolithic  architecture  to  satisfy  a  mission  goal  for  performance. 

Consequences.  This  anti-pattern  was  observed  in  three  of  the  six  programs  under 
study.  In  the  program  in  which  it  was  most  visible,  a  new  system  with  significant  new 
capabilities  was  needed;  a  business  goal  was  to  replace  several  systems  that  were  “end-of- 
life.”  The  acquisition  strategy  specified  a  slow,  deliberate  pace  to  ensure  that  the  new 
capability  was  defined  correctly.  But  ignored  in  this  strategy  was  the  urgent  need  by  users  to 
replace  these  failing  systems  as  quickly  as  possible.  When  the  operators  and  maintainers  of 
the  legacy  systems  became  aware  of  the  intended  acquisition  strategy,  they  forced  a  major 
change  in  focus  for  the  program.  The  consequence  was  a  significant  delay  for  the  program. 

In  the  other  programs  where  this  anti-pattern  was  observed,  the  effect  was  less 
pronounced.  Systems  were  delivered,  although  with  less  functionality  than  was  expected.  In 
all  cases,  follow-on  programs  have  been  started  to  create  the  functionality  that  was  originally 
promised  by  these  programs. 

In  sum,  the  overall  effect  of  this  anti-pattern  is  that  important  guiding  documents — in 
particular,  the  architecture  and  acquisition  strategy,  or  both — fail  to  reflect  the  joint  influence 
of  both  business  and  mission  goals.  Inevitably,  the  lack  of  documentation  of  missing 
business-related  goals  (and  their  associated  quality  attributes  for  the  architecture)  will  result 
in  a  system  that  fails  to  meet  the  expectations  of  at  least  some  of  the  key  stakeholders, 
because  the  joint  expectations  of  all  of  the  stakeholders  have  never  been  adequately 
reasoned  about. 

Unresolved  Conflicting  Goals 

Context.  This  anti-pattern  is  often  a  direct  consequence  of  the  previous  one,  the 
distinction  being  that  the  first  anti-pattern  refers  to  the  absence  of  well-documented  business 
goals,  while  this  one  refers  to  the  lack  of  an  analysis  and  de-confliction  of  the  known  goals. 

Problem.  The  variety  and  scope  of  mission  and  business  goals  can  be  very  large, 
and  for  a  program  of  any  consequence,  there  will  likely  be  conflicts  among  some  of  these 
diverse  goals  and  priorities.  One  factor  that  compounds  the  problem  is  that  the  business 
goals  and  the  mission  goals  are  often  developed  by  people  from  different  communities — 
people  with  very  different  concerns. 

But  to  reason  about  these  conflicts  requires  that  all  of  these  goals  be  considered 
jointly,  so  that  their  mutual  influence  can  be  understood  and  misalignments  negotiated. 
Reasoning  is  obviously  impossible  if,  as  in  the  previous  anti-pattern,  the  business  goals  are 
not  well  documented.  But  even  if  there  is  a  set  of  well-documented  business  goals,  no 
processes  or  criteria  exist  by  which  tradeoffs  between  important  business  and  mission  goals 
can  be  made.  It  is  often  not  even  clear  who  should  arbitrate  such  goal  conflicts. 

A  frequently  observed  example  of  this  anti-pattern  is  reflected  in  the  conflict  between 
the  goal  of  introducing  a  new  or  updated  system  and  the  additional  goal  of  avoiding 
impacting  how  current  end  users  perform  their  tasks.  At  best,  this  is  a  conflict  of 
expectations  that  is  not  fully  understood  until  the  system  is  deployed,  potentially  presenting 
a  barrier  to  deployment. 

Finally,  one  specific  problem  that  is  often  observed,  and  one  that  mixes  both 
business  and  mission  elements,  is  that  a  program  shares  dependencies  with  other  systems 
in  a  larger  SoS  context.  Too  often,  each  of  these  systems  is  considered  in  isolation,  and  the 
mission  and  business  effects  that  each  program  should  have  on  the  others  is  ignored.  When 
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these  effects  surface,  joint  consideration  is  often  carried  out  too  late  in  program  execution  to 
be  effective  or  to  succeed. 

Observed  Response  to  the  Problem.  Program  personnel  tend  to  separate  into 
business  (e.g.,  acquisition  strategy)  and  mission  (e.g.,  system  and  software  architecture) 
communities,  each  of  which  tends  to  work  in  isolation.  Given  this  separation,  these 
personnel  tend  to  produce  artifacts  that  reflect  the  goals  and  priorities  known  to  them;  these 
in  turn  may  be  misaligned  in  that  they  reflect  unresolved  conflicts  between  business  and 
mission  goals  such  as  those  described  previously. 

Consequences.  This  anti-pattern  was  observed  in  five  of  the  six  programs  under 
study.  In  one  program,  various  business  goals  concerning  reuse,  using  multiple  contractors, 
reducing  integration  costs,  and  such  mission  goals  as  greatly  increased  performance  had 
produced  numerous  unresolved  conflicts.  When  the  gravity  of  the  conflicts  was  belatedly 
understood  (by  an  independent  “tiger  team”),  both  the  initial  acquisition  strategy  and  the 
initial  architecture  were  abandoned,  and  a  major  reconsideration  of  both — in  which  they 
would  be  reconciled  and  aligned — was  begun.  While  this  brought  about  a  significant  delay,  it 
avoided  the  far  worse  result  of  a  system  that  failed  both  its  business  and  mission 
stakeholders. 

In  general,  it  is  likely  that  in  a  program  of  any  consequence,  conflicts  between 
business  and  mission  goals  will  exist.  But  if  the  conflicts  are  not  reconciled  before  the 
acquisition  strategy  and  the  architecture  (both  system  and  software)  are  defined,  the 
negative  effects  that  these  conflicts  will  have  will  be  large.  Unless  joint  consideration  of 
mission  and  business  goals  is  carried  out  early  in  program  execution,  conflicts  between 
goals  will  soon  become  difficult,  or  even  impossible,  to  reconcile.  And,  ultimately, 
stakeholders  who  expected  their  mission  or  business  goals  to  be  reflected  in  the  acquisition 
strategy  and  then  satisfied  by  the  software  architecture  are  unhappily  surprised  when  the 
system  cannot  support  their  mission  or  business  objectives. 

Failure  to  Adapt 

Context.  This  anti-pattern  often  occurs  when  program  duration  is  very  long.  The 
reasons  for  length  can  be  inherent,  such  as  when  a  system  is  unprecedented  and  requires 
considerable  time  to  solve  massive  engineering  problems  (for  instance,  creation  of  the  Joint 
Strike  Fighter).  Or  the  reasons  for  highly  extended  program  duration  can  be  circumstantial, 
such  as  a  protracted,  complex  protest  to  a  contract  award.  This  anti-pattern  can  also  occur 
when  a  program  evolves  from  providing  limited  capabilities  to  providing  a  much  wider  range 
of  capabilities. 

Problem.  In  most  programs,  both  the  acquisition  strategy  and  the  architecture  are 
optimized  to  meet  the  goals  and  priorities  that  exist  at  the  start  of  the  program.  However, 
goals  and  priorities  naturally  evolve  overtime:  Examples  of  such  change  could  include  the 
need  to  combat  new  and  unexpected  threats,  or  a  desire  to  modernize  a  capability  using 
new  technology.  The  essential  problem  lies  in  the  fact  that  the  architecture  and  acquisition 
strategy  that  are  initially  defined  may  not  be  flexible  enough  to  respond  to  these  changes 
without  a  good  amount  of  revision  and  redefinition. 

Further,  and  compounding  the  problem,  there  are  no  widely  applicable  processes  for 
rapidly  revising  acquisition  strategy  for  changed  business  goals,  nor  are  there  widely  used 
processes  to  accommodate  changes  to  architecture  as  a  result  of  such  changed  goals. 

Observed  Response  to  the  Problem.  When  such  changes  as  those  described 
previously  occur,  program  personnel  are  often  unsure  about  whether  the  architecture, 
acquisition  strategy,  or  both  can  accommodate  the  needed  changes  and,  even  if  they  can, 
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whether  the  changes  can  be  accomplished,  given  the  time  and  effort  that  will  be  required 
(e.g.,  to  get  all  necessary  approvals  for  a  revised  acquisition  strategy).  Hence,  programs 
tend  to  continue  executing  as  though  there  has  been  minimal  change  to  the  initial  goals  and 
priorities;  there  is  little  impetus  to  revise  the  acquisition  strategy  or  make  anything  more  than 
minimal  alterations  to  the  architecture.  In  effect,  the  program  is  operating  with  either  an 
implicit  change  in  acquisition  strategy  or  a  mismatch  between  the  architecture  defined 
initially  and  the  changed  mission  goals,  or  both. 

Consequences.  In  one  of  the  programs  we  studied,  this  anti-pattern  had  a 
considerable  negative  effect.  The  program  initially  had  a  very  successful  architecture  and 
acquisition  strategy;  the  program  was  so  successful  that  its  scope  grew  from  delivering 
capability  for  one  system  to  delivering  capability  to  multiple  systems  of  a  similar  type.  The 
architecture  and  acquisition  strategy  survived  this  change  in  scope  initially,  but  as  the 
separate  systems  matured,  and  as  need  arose  for  them  to  interoperate,  unexpected 
demands  were  placed  on  the  architecture.  An  additional  factor  was  that  the  stakeholders 
became  increasingly  interested  in  system  reliability — a  new  quality  attribute  for  this  program. 
At  this  point,  both  the  architecture  and  the  acquisition  strategy  failed.  This  program  has 
entered  a  strategic  pause  while  the  architecture  and  the  acquisition  strategy  are 
reconsidered  in  light  of  both  the  new  requirement  (reliability)  and  the  increased  complexity  of 
what  is  now  a  SoS. 

In  general,  this  anti-pattern  reflects  a  natural  tendency  to  stay  the  course,  even  when 
circumstances  change  and  external  conditions  evolve. 

Turbulent  Acquisition  Environment 

Context.  This  anti-pattern  is  closely  related  to  anti-pattern  #3  (Failure  to  Adapt).  But 
in  this  case,  the  cause  is  not  extended  program  evolution  but  rather  severe  instability  in 
multiple  program  elements.  This  instability  is  manifest  by  changes  in  goals,  strategy,  or 
architecture  that  are  so  frequent  and  contradictory,  they  require  adaptation  that,  even  under 
the  best  circumstances,  the  program  is  unable  to  accommodate.  These  changes  can  be 
political,  strategic,  technological,  or  fiscal. 

Problem.  Several  causes  can  bring  about  program  turbulence.  Budgets  can  undergo 
major  revisions,  and  major  portions  of  a  program’s  funding  can  be  withdrawn.  Mission 
circumstances  can  be  suddenly  changed,  or  radically  new  technologies  can  be 
disseminated  rapidly.  Programs,  particularly  if  they  are  perceived  to  be  in  severe  difficulty, 
can  face  significant  revisions  of  goals  and  purpose.  Joint  programs  often  undergo  periodic 
management  shifts  when  different  services  assume  primary  responsibility. 

In  cases  where  one  or  more  of  the  previously  mentioned  conditions  are  present,  a 
program  can  find  itself  faced  with  the  demand  for  change  that  is  impossible  to 
accommodate.  The  magnitude  of  the  requested  change  is  often  unrealistic,  impractical,  or 
impossible,  given  time  and  resource  realities. 

As  the  program  personnel  attempt  to  adapt  to  the  changes,  the  original  architecture 
and  acquisition  strategy  may  now  be  highly  unsuited  to  the  changed  conditions  that  have 
been  levied  on  the  program.  The  program  falls  into  a  mode  of  “architecture  of  the  day”  or 
“acquisition  strategy  of  the  day.”  Equally  problematic  and  an  important  part  of  the  problem  is 
that  the  program  is  usually  still  contractually  held  to  some  part  of  the  original  acquisition 
strategy. 

Observed  Response  to  the  Problem.  The  frequent  and  significant  changes  in 
mission  or  business  goals  overpower  the  ability  of  the  acquisition  strategy  or  architecture  (or 
both)  to  accommodate  them.  Thus,  the  original  acquisition  strategy  is  implicitly  abandoned 
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but  without  having  a  well-defined  new  strategy  created.  Or  the  architecture  is  stretched  to 
the  breaking  point  and  loses  all  relation  to  the  original  acquisition  strategy. 

Yet  many  programs  attempt  to  continue  executing  with,  at  most,  minimal  explicit 
revision  of  the  acquisition  strategy  and/or  architecture.  The  necessary  task  to  revise  the 
acquisition  strategy  to  fully  account  for  the  changed  goals  is  seldom  performed,  nor  is  the 
work  needed  to  revise  the  original  architecture  carried  out  as  carefully  as  is  required. 

Consequences.  This  anti-pattern  was  observed  in  three  of  the  six  programs  under 
study.  In  one  of  them,  significant  changes  to  mission,  architecture,  hardware,  and  program 
direction  each  occurred,  some  repeatedly.  For  instance,  the  program  began  with  a  strong 
research  basis  but  very  quickly  was  given  mandates  to  field  equipment  as  quickly  as 
possible.  Different  quality  attributes  were  given  different  priorities  as  the  architecture  evolved 
through  several  iterations  of  development.  Different  contractors  were  given  conflicting 
priorities  throughout  program  execution.  The  result  was  that  the  program  fielded  a  small 
fraction  of  what  was  originally  planned,  after  which  the  program  was  cancelled. 

There  is  little  doubt  that  the  environment  of  DoD  acquisition  always  has  some 
instability;  that  is  the  natural  condition  of  government  programs.  But  in  the  final  analysis, 
when  the  environment  is  truly  turbulent  (i.e. ,  when  this  anti-pattern  is  strongly  present),  the 
best  result  is  likely  to  be  systems  that  are  poorly  fitted  to  the  purposes  for  which  they  are  to 
be  used.  In  the  worst  case,  they  may  even  be  unfit  for  use  at  all. 

Poor  Consideration  of  Software 

Context.  This  anti-pattern  occurs  when  critical  decisions  are  made,  especially  early 
in  a  program,  that  have  strong  negative  implications  on  the  system’s  software.  There  is  a 
historical  basis  for  this  behavior:  For  decades,  the  DoD  acquired  systems  that  were  primarily 
hardware.  But  while  the  role  and  importance  of  software  has  grown  significantly  in  recent 
years,  the  traditions  and  habits  of  acquisition  still  reflect  the  earlier,  hardware-centric 
posture. 

Problem.  Very  often,  software  is  not  deemed  a  critical  factor  in  decisions  made  at 
the  earliest  stages  of  a  program.  These  decisions  generally  are  made  with  little  or  no 
understanding  of  how  software  must  be  accounted  for  in  the  acquisition  strategy  or  the 
architecture  (or  both). 

One  symptom:  Contracts  are  organized  based  primarily  on  the  system  architecture 
and  fail  to  take  into  consideration  the  very  sizable  role  of  software.  Assumptions  are  made 
about  the  expected  integration  of  software  entities  that  are  created  separately;  such 
assumptions  are  often  made  with  no  understanding  of  the  difficulties  that  arise  when 
complex  independent  software  systems  must  interoperate  or  be  integrated.  This  leads  to 
system  architectures  and  acquisition  strategies  that  over-constrain  the  yet-to-be-defined 
software  architecture,  thus  adding  significant  complexity  to  software  development  and 
integration.  Even  in  a  software-only  system,  the  real  difficulties  that  the  software  can  pose 
(e.g.,  integration  of  many  heterogeneous  components)  are  largely  ignored. 

One  other  such  symptom  is  that  quality  attributes  that  are  important  to  the  system 
engineering  community  may  be  quite  different  from  those  that  are  significant  to  the  software 
community.  Even  if  the  two  communities  speak  of  a  single  quality  attribute  (e.g.,  reliability), 
they  refer  to  different  things.  Thus,  early  decisions  about  quality  attributes  typically  are 
decisions  about  system,  not  software,  quality  attributes. 

Observed  Response  to  the  Problem.  As  a  result,  the  acquisition  strategy  either  is 
created  with  a  strong  focus  on  system  architecture  or,  in  software-only  systems,  fails  to 
address  the  software  architecture  satisfactorily.  In  the  former  case,  this  inevitably  produces 
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a  large  gap  between  the  system  and  software  architectures;  in  the  latter  case,  the  planned 
software  acquisition  strategy  is  unrealistic  and  impractical. 

A  common  “solution”  is  that  the  software  architect  tries  to  play  “catch  up”  and  fit  the 
software  to  the  system  architecture.  Another  “solution”  is  to  ignore  the  eventual  complexities 
of  integration  and  expect  that  they  can  be  resolved  later.  In  these  cases,  the  result  is  often 
that  the  system  constraints  force  software  choices  that  are  suboptimal  for  the  whole  system. 

Consequence.  This  anti-pattern  was  observed  in  three  programs.  In  one  program, 
two  critical  early  decisions  about  software  were  made  with  very  little  understanding  of 
software’s  inherent  complexity.  The  system  was  large  and  complex,  with  three  major 
software  components.  An  early  decision  concerning  the  integration  of  those  three 
components  downplayed  all  of  the  details  of  that  integration:  who  would  do  it,  what 
resources  it  would  need,  and  how  difficult  it  would  be.  No  attempt  was  made  to  base  this 
decision  on  expert  software  advice,  nor  on  data  readily  available  from  comparable  programs 
about  the  difficulty  of  such  a  task.  Another  decision  related  to  the  assumption  that  the 
system  could  later  be  made  to  interoperate  with  another  complex  software  system.  But  no 
rigorous  assessment  was  made  of  the  difficulty  of  accomplishing  that  interoperation  nor  of 
the  resources  that  this  integration  would  require.  In  both  cases,  the  assumptions  proved 
false,  and  the  program  was  cancelled. 

In  general,  in  the  presence  of  this  anti-pattern,  there  will  be  major  software 
requirements  that  cannot  be  satisfied.  In  a  system  with  both  hardware  and  software 
elements,  this  is  often  a  direct  result  of  a  software  architecture  that  had  to  be  made  to  fit 
poorly  into  a  system  architecture  that  was  already  defined.  Further,  the  problems  and  delays 
that  result  from  the  gap  between  the  system  and  the  software  architectures  are  typically 
blamed  on  the  software  components  alone.  In  the  worst  cases,  these  suboptimal  decisions 
are  reflected  in  system-level  schedule  delays  and  cost  overruns. 

Inappropriate  Acquisition  Strategies 

Context.  The  time  factor  figures  prominently  in  DoD  acquisitions.  Starting  from  the 
earliest  moments  of  a  program  (i.e.,  the  awareness  of  a  need  for  a  new  or  updated  system), 
one  urgent,  yet  often  unstated,  imperative  is  to  move  quickly  to  avoid  any  eventualities  that 
might  delay,  or  even  prevent,  a  program  from  achieving  its  desired  milestones.  This 
imperative  is  often  exacerbated  by  the  need  to  spend  an  amount  of  money  that  was 
established  as  many  as  two  years  previously,  at  a  point  where  little  was  actually  known 
about  the  realities  that  the  program  would  face.  It  is  further  exacerbated  by  the  lengthy 
review  process  before  a  new  contract  can  be  awarded.  These  realities  of  the  time  factor 
have  led  to  acquisition  strategies  that  are  often  poorly  suited  to  an  individual  program’s 
needs. 


Problem.  There  are  multiple  causes  of  this  anti-pattern.  Program  offices  might 

•  wish  to  avoid  protests 

•  get  quick  approval  for  a  program  (e.g.,  before  anticipated  budget  cuts) 

•  lack  sufficient  acquisition  expertise  to  develop  an  acquisition  strategy  that  will 
quickly  gain  approval 

•  have  a  particular  acquisition  strategy  imposed  by  a  higher  authority 

In  another  scenario,  some  key  business  goals  (e.g.,  split  an  acquisition  that  is 
conceptually  a  single  system  into  multiple  acquisitions  to  avoid  a  “big  bang”)  are  in  direct 
conflict  with  the  technology  to  be  used,  the  system  to  be  built,  or  both. 
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Note  that  these  causes  can  be  either  external  (protests,  budget  cuts)  or  internal 
(inexperience  of  a  program  management  office  to  develop  and  defend  a  solid  acquisition 
strategy). 

Observed  Response  to  the  Problem.  Whatever  the  particular  cause,  the  result  is 
that  the  primary  goal  of  the  program  office  and  the  source  selection  team  is  to  get  through  a 
competition  and  issue  a  contract  as  quickly  as  possible.  And  even  though  the  chosen 
acquisition  strategy  might  have  a  poor  fit,  or  even  a  misfit,  with  the  business  and/or  mission 
goals  for  the  system,  the  inappropriate  acquisition  strategy  is  put  in  place.  The  program 
begins  execution,  deferring  or  ignoring  the  parts  of  the  strategy  that  have  a  poor  fit  with  the 
real  needs  of  the  system  to  be  built  and,  too  often,  the  wrong  contract  relationship  with  the 
software  developer(s). 

Consequences.  This  anti-pattern  was  observed  in  five  of  the  six  programs  under 
study.  In  one  program,  the  fear  of  a  “big  bang”  approach  led  to  splitting  the  intended  large 
system  into  two  separate  acquisitions,  with  the  assumption  that  the  two  systems  could  later 
be  integrated  into  a  large  SoS.  But  the  second  program  suffered  a  very  long  and  complex 
protest  and  did  not  get  underway  until  several  years  after  the  first  had  begun.  By  that  time, 
the  first  had  completed  most  of  its  requirements  and  was  nearing  initial  operational 
deployment.  But  there  had  been  no  input  from  the  stakeholders  of  the  second  system  about 
the  interfaces  that  would  be  needed  to  make  eventual  integration  of  the  two  systems 
feasible.  Thus,  the  plan  for  integrating  the  two  systems  was  abandoned,  and  the  second 
program  was  eventually  cancelled.  The  first  program  fielded  a  system  that  provided  only  a 
small  portion  of  the  expected  functionality. 

In  general,  in  the  presence  of  this  anti-pattern,  one  of  two  immediate  consequences 
will  emerge.  Either  the  acquisition  strategy  is  ignored  as  the  program  unfolds  or  the  program 
is  forced  to  bend  the  needs  of  the  system  to  the  inappropriate  strategy.  In  the  latter  case, 
the  system  can  reach  a  point  where  it  can  no  longer  meet  its  mission  goals,  business  goals, 
or  both. 


Overlooking  Quality  Attributes 

Context.  In  the  earliest  stages  of  a  program’s  life,  there  may  be  no  formal  program 
office  and  only  minimal  accompanying  funding  to  perform  necessary  work.  Further,  there  is 
significant  pressure  to  rapidly  produce  the  acquisition  strategy  and  initial  architecture  in 
order  for  the  program  to  be  funded.  But  there  is  no  requirement  to  use  quality  attributes  to 
define  that  architecture,  and  little  incentive  to  do  so.  Further,  in  many  cases,  the  detailed 
business  goals  are  unwritten  (see  anti-pattern  #1)  or  the  importance  of  the  software  is 
ignored  (see  anti-pattern  #5),  and  hence,  there  is  little  opportunity  to  expose  the  quality 
attributes  that  the  system  is  expected  to  manifest. 

Problem.  The  program  overlooks  the  software  quality  attributes  that  should  support 
the  goals,  whether  mission  or  business.  Instead,  programs  rely  on  key  performance 
parameters  (KPPs),  which  are  not  broken  down  in  sufficient  detail  that  architects  can  reason 
about  the  necessary  alignment  among  the  software  architecture,  system  architecture, 
acquisition  strategy,  and  aggregate  set  of  goals  for  the  system. 

Observed  Response.  In  order  to  meet  the  reporting  needs  and  get  to  a  “go”  or  “no 
go”  decision  for  a  system,  programs  put  their  engineering  resources  into  eliciting  and 
capturing  the  functional  capabilities  and  requirements  and  provide  only  minimal  attention  to 
quality  attributes  by  focusing  on  a  limited  set  (e.g.,  performance,  availability).  Worse,  the 
notions  of  those  quality  attributes  are  more  often  those  understood  by  system  engineers 
rather  than  software  engineers. 
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As  a  result,  architectural  decisions  that  should  be  based  on  extensive  consideration 
of  all  quality  attributes — software  as  well  as  systems — are  made  by  “gut  feel,”  or  by  adopting 
an  architecture  from  a  similar  or  idealized  system,  rather  than  by  reasoning  that  placed  real 
importance  on  the  specific  software  quality  attributes  for  the  system  at  hand. 

Consequences.  This  anti-pattern  was  observed  in  four  of  the  programs  under  study. 
In  one  joint  program,  the  system  was  to  be  integrated  into  operations  in  each  of  the  military 
Services.  However,  the  concept  of  operations  for  each  of  the  Services  was  different  (i.e., 
where  the  system  would  be  hosted,  security  needs,  what  other  systems  would  be 
integrated),  and  these  differences,  all  of  which  implied  different  quality  attributes,  were  not 
recognized  early  in  the  program.  Neither  the  acquisition  strategy  nor  the  architecture 
accommodated  these  differences.  The  quality  attribute  descriptions  that  were  constructed 
reflected  the  needs  of  only  one  of  the  Services  and  focused  only  on  technical  issues;  they 
explicitly  ignored  that  same  Service’s  business  goals.  It  eventually  became  apparent  that  the 
program  office  and  the  end  users  had  very  different  approaches  to  meeting  even  the  single 
Service’s  stated  needs. 

In  general,  the  primary  consequence  of  this  anti-pattern  is  that  the  resultant  system 
architecture  (and  the  likely  inefficient  software  architecture)  will  satisfy  only  some  of  the 
goals;  others  will  be,  at  best,  partially  satisfied  and  often  unsatisfied.  In  the  longer  term, 
since  sufficient  knowledge  of  the  quality  attributes  is  lacking,  the  program  will  not  have  the 
strong  analytic  base  needed  to  fully  understand  the  impacts  of  different  modes  of  evolution 
that  might  be  needed  over  the  system’s  life  cycle. 

Countering  the  Patterns 

The  anti-patterns  described  in  the  preceding  section  provide  evidence  of  undesirable 
behaviors  that  are  repeated  across  multiple  programs.  We  believe  that  these  undesirable 
behaviors  are  not  intentional  but  indicate  flaws  in  the  existing  approach  to  the  acquisition  of 
software-reliant  systems.  We  also  believe  that  at  least  part  of  the  solution  to  this  problem 
lies  in  analyzing  how  these  anti-patterns  operate  at  both  the  micro  level  and  the  macro  level. 
In  the  previous  section,  we  examined  the  individual  consequences  of  each  anti-pattern.  In 
this  section,  we  consider  how,  in  combination,  they  jointly  can  affect  the  major  entities  that 
participate  in  the  acquisition  process. 

Necessary  Entities  and  Artifacts 

The  anti-patterns  we  observed  relate  to  a  small  number  of  distinct  entities,  each  of 
which  has  major  importance  for  a  program  and  the  system  it  is  building.  For  instance,  one 
anti-pattern  focused  on  how  programs  ignored  the  impact  of  quality  attributes.  There  is 
ample  evidence  and  experience,  such  as  that  discussed  by  Gagliardi,  Wood,  Morrow,  and 
Klein  (2010),  showing  that  the  main  drivers  for  software  architecture  should  be  the  quality 
attributes  that  the  system  must  exhibit.  These  quality  attributes  are  derived  from  another  key 
entity,  the  mission  needs  expressed  by  stakeholders. 

Another  anti-pattern  dealt  with  business  goals  that  were  either  only  expressed 
generally  or  only  implicitly  understood.  The  business  goals  for  a  program  are  an  additional 
key  entity.  But  these  are  likely  the  goals  of  a  very  different  set  of  stakeholders.  Although  not 
commonly  understood,  the  business  goals,  like  the  mission  goals,  will  have  quality  attributes 
that  should  be  the  main  drivers  for  the  acquisition  strategy;  we  assert  that  these  other  quality 
attributes  are  as  important  as  those  derived  from  the  mission  goals.  We  will  refer  to  these 
other  quality  attributes  as  acquisition  quality  attributes. 

Given  these  two  sets  of  goals,  which  in  turn  are  the  principal  drivers  for  two  very 
critical  entities  (i.e.,  the  acquisition  strategy  and  the  software  and  system  architectures),  the 
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potential  for  conflicts  between  those  goals  is  large,  as  is  shown  in  the  anti-pattern  of 
conflicting  goals. 

Finally,  the  notion  of  “the  stakeholders”  embodies  a  complex  and  diverse  collection 
of  individuals  and  organizations;  they  are  the  sources  for  the  goals  and  the  recipients  of  the 
benefits  of  the  system  to  be  created. 

We  posit  several  key  entities  of  interest: 

•  mission  goals 

•  the  (mission)  quality  attributes  implicit  in  those  goals 

•  business  goals 

•  the  (acquisition)  quality  attributes  implicit  in  those  goals 

•  the  acquisition  strategy 

•  the  software  and  system  architectures,  which  are  closely  related,  but 
separate 

•  the  different  sets  of  stakeholders  who  have  expressed  needs  that  are 
captured  by  the  mission  and  business  goals 

While  these  entities  represent  a  diverse  set  of  things,  including  humans 
(stakeholders)  and  intangibles  (goals),  we  posit  that  all  of  these  entities  must,  at  least  in 
some  manner,  be  manifest  in  physical  artifacts.  Some  such  artifacts  are  immediately 
obvious:  the  mission  goals  will  ultimately  be  reflected  in  a  requirements  specification;  an 
acquisition  strategy  document  is  mandatory  for  any  program.  Other  artifacts  are  less  well 
defined  and  may  not  even  be  present  in  a  given  program.  We  assert  that  they  are  just  as 
necessary. 

The  stakeholders  for  a  given  acquisition,  for  instance,  cannot  be  a  vague  collection 
of  unknown  persons  but  must  be  defined  with  at  least  minimal  specificity:  “the  HR  personnel 
who  do  data  entry  for  the  Air  Force  Logistics  Command,”  “the  Assistant  Secretary  of  the 
Navy  for  XYZ,”  and  so  forth.  The  definition  of  the  stakeholders  may  not  have  a  formal 
document  type  associated  with  it,  but  it  must  be  physical:  There  must  be  some  way  to 
determine  who  precisely  has  one  or  another  goal,  if  only  to  assist  in  determining  priorities 
and  negotiating  conflicts.  Similarly,  the  business  goals  themselves  may  first  be  only  general 
expressions  found  in  a  statement  of  need.  But  eventually  those  must  find  some  form  of 
detailed  notation  in  a  physical  document  that  is  a  real  analog  to  a  requirements 
specification.  Thus,  we  assert  that  each  of  the  entities  listed  previously  has  an  associated 
artifact,  which  we  can  inspect,  reason  about,  and  compare  with  other  artifacts  of  the 
acquisition  process. 

Necessary  Relationships 

These  entities  are  related  to  each  other  by  means  of  several  different  relationships. 
For  each,  we  use  the  formula  “Entity  X  <relationship>  Entity  Y.”  The  following  are  the 
pertinent  relationships  identified  in  our  analysis: 

•  <have> 

•  <are  embodied  by> 

•  <are  consistent  with> 

•  <drive> 
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•  <constrains> 


•  <informs> 

We  show  these  entities  and  relationships  in  Figure  1.  Some  relationships  are 
unidirectional,  and  the  arrow  indicates  which  entity  is  the  actor  (e.g.,  for  “X  <drive>  Y,”  the 
direction  of  the  arrow  is  from  X  to  Y).  Some  of  these  relationships  are  reflexive  (e.g.,  “X 
<informs>  Y”  and  “Y  <informs>  X”).  In  these  cases,  the  arrow  is  two-headed. 


satisfies 


Figure  1.  Key  Entities  and  Relationships  Identified  During  Our  Analysis 


When  we  consider  these  entities  and  relationships  in  light  of  the  anti-patterns,  we 
see  the  following: 

The  <have>  relationships  concern  anti-patterns  #1  and  #4.  While  stakeholders 
have  business  goals,  these  goals  are  often  not  expressed;  the  problem  is  exacerbated  by 
the  lack  of  a  process  for  recording  business  goals.  If  these  goals  can  be  captured  so  that  the 
collection  of  business  goals,  stemming  from  all  stakeholders,  exists  in  a  coherent  document, 
then  anti-pattern  #1  cannot  occur;  the  business  goals  will  have  been  documented.  If  a 
program  office  captures  and  records  these  goals,  then  the  office  can  reason  about  changes 
in  the  acquisition  environment.  It  can  determine  whether  the  inevitable  changes  can  be 
accommodated  within  the  program’s  current  scope  or  whether  a  reset  of  the  acquisition 
strategy  or  the  architectures  or  both  will  be  required.  If  a  program  office  can  perform  such 
reasoning  then,  although  turbulence  in  the  acquisition  environment  cannot  be  prevented,  the 
program  office  can  have  an  appropriate  response  to  the  turbulence  and  prevent  the 
occurrence  of  anti-pattern  #4. 

The  Ore  embodied  by>  relationships  concern  anti-pattern  #7.  If  the  business 
and  mission  goals  are  analyzed  and  re-expressed  in  terms  of  acquisition  quality  attributes 
and  software/system  quality  attributes,  respectively,  then  anti-pattern  #7  cannot  occur  since 
the  quality  attributes  will  have  been  carefully  analyzed  as  part  of  program  creation. 
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The  Ore  consistent  with>  relationship  concerns  anti-pattern  #2.  Assuming  that 
all  of  the  relevant  quality  attributes  (software,  system,  and  acquisition)  have  been  clearly 
expressed,  it  will  be  possible  to  reason  about  all  quality  attributes,  comparing  and 
performing  tradeoffs  between  the  types  of  attributes.  If  the  quality  attributes  are  consistent 
with  each  other,  then  conflicts  among  the  goals  will  have  been  resolved  and  anti-pattern  #2 
will  not  occur. 

The  <drive>  relationships  concern  anti-patterns  #5  and  #6.  Years  of  research 
and  practical  usage  of  quality  attributes  have  demonstrated  that  they  should  be  the  primary 
influences  on  both  software  and  system  architectures  (Bass,  Clements,  &  Kazman,  2012).  In 
the  situations  where  the  quality  attributes  are  used  to  create  the  architectures,  then  we  can 
be  certain  that  those  qualities  important  to  the  program  have  been  considered  and  the 
resultant  architecture  is  consistent  with  the  quality  attributes.  In  such  a  case,  anti-pattern  #5 
cannot  occur.  Similarly,  if  the  acqusition  strategy  is  derived  from  the  quality  attributes  that 
have  been  developed  from  the  business  goals,  then  it  is  likely  that  the  strategy  will  indeed 
reflect  all  of  the  stakeholder  expectations  and  cannot  be  considered  to  be  inappropriate  to 
the  institutional  goals  of  the  organizations  involved,  thus  preventing  the  occurrence  of  anti¬ 
pattern  #6. 

The  <constrains>  relationship  concerns  anti-pattern  #3.  Even  when  the 
acquisition  strategy  and  the  software  architecture  are  specifically  aligned,  this  alignment 
must  be  maintained  through  the  life  of  the  system  and/or  the  life  of  the  program  office.  As 
the  system  matures,  new  goals  emerge  that  must  be  accommodated  in  the  acquisition 
strategy  or  the  architecture  or  both.  Ensuring  that  the  architectures  continue  to  constrain  the 
acquisition  strategy  and  the  acquisition  strategy  continues  to  constrain  the  architectures  will 
increase  the  likelihood  that  the  program  is  feasible.  In  such  a  way,  anti-pattern  #3  can  be 
prevented  from  occurring. 

The  <informs>  relationship  does  not  specifically  concern  an  observed  anti¬ 
pattern.  Nonetheless,  considerable  research  has  shown  the  importance  of  mutual  influence 
between  software  and  system  architecture.  Maier  (1998)  succinctly  described  the  different 
perspectives  and  the  impact  of  not  explicitly  balancing  the  engineering  and  managing 
between  these  two  architectures.  We  therefore  include  it  here  for  completeness. 

Figure  2  summarizes  the  anti-patterns  as  they  affect  the  entities  and  relationships. 
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6.  Inappropriate 
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Figure  2.  Anti-Patterns  That  Affect  Specific  Relationships  and  Entities 
Acquisition  Quality  Attributes 

A  key  aspect  in  our  work  is  the  notion  of  acquisition  quality  attributes.  This 
represents  a  new  concept  and  is  the  focus  of  our  current  investigations.  We  have  introduced 
them  because  we  are  motivated  by  the  considerable  body  of  work  on  software  quality 
attributes  and  expect  that  the  acquisition  quality  attributes  are  as  important  to  acquisition 
strategy  as  software  quality  attributes  are  to  the  software  architecture.  Because  we  see  such 
a  close  analogy  between  these  two  types  of  quality  attributes,  we  believe  that  research  into 
these  attributes  can  be  modeled  on  the  research  into  software  quality  attributes. 

Role  of  Acquisition  Quality  Attributes 

Acquisition  quality  attributes  are  derived  from  the  business  goals  and  exert  their 
strongest  influence  on  the  acquisition  strategy.  They  relate  the  business  goals  to  the 
acquisition  strategy  in  the  same  way  that  the  software  quality  attributes  relate  the  mission 
goals  to  the  software  architecture.  To  formulate  acquisition  quality  attributes  for  a  program 
implies  the  following  step  of  explicitly  developing  an  acquisition  strategy  that  accommodates 
them:  They  must  be  designed  into  the  strategy  to  minimize  the  risk  of  failure.  Note  that 
incorporating  these  qualities  into  the  acquisition  strategy  is  no  guarantee  that  the  program 
will  be  successful,  but  it  lays  the  foundation  upon  which  success  can  be  built. 

As  with  software  quality  attributes,  acquisition  quality  attributes  will  be  dependent  on 
the  specific  circumstances  of  the  program  in  question.  For  example,  a  program  may  have 
little  concern  for  long-term  sustainability  if  there  is  little  expectation  that  the  deployed  system 
will  be  sustained  over  the  long  term;  hence,  the  acquisition  strategy  should  be  concerned 
about  many  quality  attributes  but  not  about  sustainability.  (This  can  occur  in  situations  where 
there  is  an  imperative  need  to  get  something  deployed  as  rapidly  as  possible.).  Or  a 
program  may  be  developed  as  a  research  prototype  with  no  concern  about  deployability  or 
sustainability  if  the  major  goal  is  to  prove  a  concept,  not  field  a  capability.  The  key  point  is 
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that  these  attributes  must  be  understood  in  the  context  of  each  particular  program  so  that  a 
suitable  acquisition  strategy  can  be  developed. 

As  with  software  quality  attributes,  we  expect  that  some  acquisition  quality  attributes 
will  be  universal.  Thus,  every  program  is  started  with  some  sense  of  producing  a  good 
result,  and  we  would  expect  every  program  to  have  some  sense  of  “achievability,”  although 
precisely  what  that  means  would  differ  from  program  to  program.  Similarly,  programs  with 
the  goals  for  a  long  lifetime  will  exhibit  sustainability  and  evolvability;  if  not,  the  programs  are 
unlikely  to  enjoy  their  long  life  spans  without  major,  costly  adjustments  over  their  lifetimes. 

Giving  Meaning  to  Acquisition  Quality  Attributes 

As  stated  previously,  our  view  of  acquisition  quality  attributes  is  that  they  are  an 
analog  to  software  quality  attributes,  although  the  qualities  they  relate  to  may  differ.  And  like 
software  quality  attributes,  they  have  little  inherent  meaning  that  is  not  ambiguous  and  must 
be  given  precise  meaning  by  some  other  mechanism. 

One  approach  would  be  to  try  to  develop  precise  definitions  of  each  acquisition 
quality  attribute.  Our  experience,  however,  is  that  this  would  lead  to  the  same  difficulties 
found  with  software  or  system  quality  attributes,  where  many  different  and  inconsistent 
quality  models  exist  and  the  definitions  are  either  too  general  or  too  contradictory  to  be  of 
use.  (For  instance,  one  could  conjecture  many  definitions  of  sustainability,  but  which 
definition  was  appropriate  to  a  specific  program  and  system  would  still  need  to  be  resolved.) 

Instead,  we  follow  the  approach  used  in  software  architecture  (and  later,  system 
architecture)  of  developing  scenarios,  which  have  the  purpose  of  defining  precisely  the 
meaning  of  the  attributes  in  the  context  of  a  specific  acquisition.  Such  an  approach 
eliminates  needless  discussion  about  the  “correct”  definition  of  any  individual  attribute  and, 
more  importantly,  gives  meanings  based  on  the  context  of  a  particular  acquisition. 

The  software  architecture  work  at  the  SEI  was  of  great  benefit  in  representing  quality 
attribute  scenarios  in  six  parts:  source,  stimulus,  artifact,  environment,  response,  and 
response  measure  (Barbacci  et  al. ,  2003).  Following  this  approach,  we  characterize  the 
parts  of  an  acquisition  quality  attribute  in  the  same  manner: 

•  Source :  The  person,  group,  or  organization  that  generates  the  stimulus. 

•  Stimulus :  A  condition  or  event  that  needs  to  be  considered  when  it  occurs 
during  the  acquisition. 

•  Environment :  The  conditions  under  which  the  stimulus  occurs. 

•  Artifact  The  artifact  that  is  stimulated. 

•  Response :  The  activity  undertaken  after  the  arrival  of  the  stimulus. 

•  Response  measure :  When  the  response  occurs,  ideally  this  will  be 
measurable  in  some  fashion  so  that  the  scenario  is  testable. 

As  an  example,  in  one  program,  an  acquisition  quality  attribute  scenario  for 
“flexibility”  could  be  described  by  this  scenario:  “In  an  environment  where  the  continuing 
resolution  bans  new  program  starts,  warfighters  in  the  field  demand  new  capability  now.  The 
new  capability  is  added  to  an  existing  development  contract  without  change  in  scope.”  The 
scenario  elements  would  then  correspond  to 

Source:  Warfighters  in  the  field 

Stimulus:  ...  demand  a  new  capability  now. 
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Environment:  The  continuing  resolution  bans  new  program  starts 
Artifact:  ...  and  so  an  existing  development  contract 
Response:  ...  is  modified  to  add  the  desired  capability 
Response  measure:  ...  without  change  in  scope 

Yet  in  another  program,  the  same  quality  attribute  of  “flexibility”  could  yield  a 
somewhat  different  scenario  due  to  the  differences  between  the  programs.  For  example,  this 
other  scenario  might  be:  “In  an  environment  where  the  deployment  strategy  has  not  yet 
been  defined,  warfighters  in  the  field  demand  new  capability  now.  The  existing  development 
contract  is  modified  to  provide  an  early  delivery  of  the  desired  capability.”  Here,  the  scenario 
elements  are 

Source:  Warfighters  in  the  field 
Stimulus:  ...  demand  a  new  capability  now 

Environment:  The  deployment  strategy  has  not  yet  been  fully  defined 
Artifact:  ...  so  the  development  contract 
Response:  ...  is  modified  to  provide  the  desired  capability 
Response  measure:  ...  to  support  an  early  delivery 

In  both  of  these  cases,  the  scenarios  provide  specific  meanings  for  “flexibility”  but 
also  indicate  how  they  would  imply  different  acquisition  strategies  for  the  two  situations. 

Working  With  Acquisition  Quality  Attribute  Scenarios 

We  expect  that  the  notion  of  acquisition  quality  attributes  has  the  same  relevance  to 
the  acquisition  professional  that  software  quality  attributes  have  to  the  software  architect. 
Thus,  in  practice,  we  would  hope  that  a  valuable  exercise  for  an  acquisition  team  early  in  a 
program’s  life  would  be  to  consider  the  program  they  are  defining  in  terms  of  the  acquisition 
quality  attributes  implied  by  the  program’s  business  goals  (whether  explicitly  stated  or 
otherwise).  In  such  an  exercise,  we  start  with  the  business  goals  and,  for  each  one,  elicit 
one  or  more  acquisition  quality  attributes.  These  are  then  used  to  elicit  detailed  scenarios 
that  would  give  precise  meaning  to  the  quality  attributes.  One  of  the  most  effective 
mechanisms  for  eliciting  scenarios  is  through  guided  brainstorming  where  appropriate 
stakeholders  volunteer  scenarios  that  are  subsequently  reviewed  for  their  importance.  One 
key  issue  is  to  explicitly  consider  the  difficulty  of  satisfying  the  scenarios.  Scenarios  that  are 
both  of  high  importance  and  difficult  to  satisfy  are  the  ones  that  will  receive  the  most 
attention  during  the  subsequent  analysis. 

Again,  working  as  a  group,  the  scenarios  must  then  be  analyzed  for  consistency  with 
each  other.  If  the  consensus  is  that  some  scenarios  are  inconsistent  with  others,  this  implies 
that  some  business  goals  are  inconsistent  with  each  other  and  that  the  program  cannot 
satisfy  all  of  those  goals.  Ideally,  conflicting  business  goals  will  be  negotiated  by  the 
stakeholders  until  they  no  longer  conflict,  thus  leading  to  an  acquisition  approach  that  can 
satisfy  the  stated  goals. 

Because  acquisition  quality  scenarios  define  how  the  program  must  respond  to 
events,  the  scenarios  can  now  be  used  to  determine  whether  a  proposed  acquisition 
strategy  will  allow  the  program  to  respond  appropriately  to  expected  future  events.  The 
collection  of  scenarios  can  be  used  to  shape  the  program’s  acquisition  strategy  so  that  it  will 
embody  the  negotiated  goals  for  the  program. 
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Finally,  we  expect  that  both  the  elicitation  and  analysis  of  acquisition  quality 
attributes,  and  their  associated  scenarios,  can  be  combined  with  the  elicitation  and  analysis 
of  software  quality  attributes.  In  such  a  case,  the  consistency  analysis  can  account  for  both 
software  and  acquisition  quality  attributes,  thereby  leading  to  a  negotiation  between  mission 
and  business  goals.  Although  harmony  between  the  two  sets  of  scenarios  does  not 
guarantee  program  success,  we  posit  that  if  there  are  conflicts  between  the  software-based 
and  acquisition-based  scenarios,  it  is  extremely  likely  that  the  architecture  will  conflict  with 
the  acquisition  strategy  and  be  a  significant  detractor  to  program  success. 

We  are  currently  in  the  process  of  validating  our  assertion  that  acquisition  quality 
attributes  can  be  used  to  define  the  relationship  between  business  goals  and  acquisition 
strategies.  We  are  performing  this  validation  by  reviewing  various  programs,  extracting  their 
business  goals,  developing  plausible  scenarios,  and  then  defining  pieces  of  the  acquisition 
strategy  that  would  satisfy  the  scenario. 

In  addition,  we  are  examining  several  cases  of  long-lived  programs  where  we  can 
clearly  distinguish  between  the  original  goals  for  the  program  and  changes  to  those  goals. 
Through  interviews,  we  extracted  the  business  goals  and  the  associated  acquisition  strategy 
that  the  programs  adopted,  developing  acquisition  quality  attribute  scenarios  as  examples. 
As  new  goals  were  added,  we  developed  new  acquisition  quality  attributes.  Our  goal  with 
this  exercise  was  not  to  second-guess  or  judge  the  original  acquisition  strategy  but,  rather, 
to  show  that  scenarios  can  be  found  to  be  consistent  or  in  conflict  and  that  the  new  scenario 
can  be  used  to  judge  the  effectiveness  of  the  existing  acquisition  strategy. 

Conclusions  and  Next  Steps 

Alignment  between  the  architecture  and  acquisition  strategy  does  not  occur  naturally. 
Our  research  has  revealed  seven  behavioral  patterns,  or  anti-patterns,  that  lead  to 
misalignments,  which  have  contributed  to  program  restarts,  cancellations,  and  other  failures. 

We  have  also  learned  that  the  effect  of  these  anti-patterns  can  most  easily  be 
observed  through  several  key  entities,  and  the  relationships  that  should  hold  between  and 
among  them.  A  key  insight  derived  from  this  research  is  that  the  relationship  of  the 
acquisition  strategy  to  the  business  goals  is  analogous  to  the  relationship  between  the 
software  architecture  and  the  mission  goals.  Our  continuing  investigations  include 
developing  acquisition  quality  attributes  with  associated  six-part  scenarios  and  validating 
that  they  can  be  used  to  distinguish  between  appropriate  and  inappropriate  acquisition 
strategies. 

Our  future  goal  is  to  create  an  alignment  method  that  assists  program  office 
personnel  to  systematically  identify  probable  areas  of  mismatch  between  business  and 
mission  goals,  acquisition  strategy,  and  software  architecture.  With  such  a  method,  two 
outcomes  would  be  feasible:  (1)  program  managers  can  build  acquisition  strategies  that 
more  systematically  eliminate  one  key  cause  of  program  failure,  and  (2)  acquisition 
overseers  can  better  evaluate  the  adequacy  of  the  acquisition  strategy  as  programs  move  to 
different  life  cycle  phases.  While  this  alone  will  not  guarantee  program  success,  we  believe 
that  it  could  be  a  significant  factor  in  avoiding  failures. 
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Abstract 

Reducing  cost  and  development  time,  while  preserving  acceptable  levels  of  performance,  is  a 
priority  for  all  government-sponsored  complex  product  development.  One  avenue  for 
improving  outcomes  is  to  use  architecting  strategies  to  guide  development  decisions. 
Frequent  examples  are  commonality,  interoperability,  modularity,  flexibility,  extensibility, 
robustness,  openness,  and  adaptability.  A  second  avenue  for  improving  outcomes  is  better 
acquisition  strategies.  The  two  are  often  considered  in  isolation.  This  paper  begins  an 
examination  of  how  the  choice  of  architecting  strategy  affects  the  choice  of  acquisition 
strategy,  and  vice  versa. 

As  a  first  step,  the  paper  synthesizes  existing  literature  and  provides  straightforward 
definitions  of  each  of  the  architecting  strategies.  As  a  second  step,  the  paper  maps  each  of 
the  defined  architecting  strategies  against  two  common  axes  of  acquisition  design, 
specifically  openness  to  competition  and  sensitivity  to  requirements  change.  The  conclusions, 
while  tentative,  show  that  increasing  attention  to  the  interaction  between  how  systems  are 
designed  and  how  they  are  acquired  may  have  a  significant  effect  on  the  cost,  schedule,  and 
performance  of  complex  product  development. 

Introduction 

Reducing  cost  and  development  time,  while  preserving  acceptable  levels  of 
performance,  is  a  priority  for  complex  product  development  in  both  military  and  civilian 
systems.  One  avenue  for  improving  development  is  to  use  particular  architecting  strategies 
to  guide  high-level  development  decisions.  These  strategies  reflect  the  priorities  that  the 
customer  places  on  different  aspects  of  the  product.  Different  strategies  lead  to  different 
design  decisions  and  ultimately  different  outcomes.  For  example,  developments  based  on 
the  “robustness”  strategy  might  trade  high  performance  under  specific  conditions  for 
acceptable  performance  over  a  range  of  conditions.  This  paper  considers  common, 
modular,  open,  flexible,  adaptable,  robust,  extensible,  and  interoperable  architecting 
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strategies.  Many  of  these  strategies  have  at  one  time  or  another  been  exhorted  as  the 
solution  to  acquiring  complex  products.1 

However,  there  appears  to  be  a  disconnect  between  the  use  of  these  strategies  and 
their  effect  on  complex  product  development,  particularly  in  the  government  sphere.  New 
architecting  strategies  are  encouraged  at  the  highest  levels,  but  the  cost  of  acquiring  new 
complex  systems  continues  to  climb  (Berteau  et  al.,  2010;  Peters,  2009;  Moore,  2011). 

We  observed  two  factors  that  we  hypothesized  to  be  likely  contributors  to  the  lack  of 
impact  of  these  architecting  strategies  on  project  acquisition  costs.  The  first  factor  is  a  lack 
of  effective  communication  about  what  the  architecting  strategies  mean,  caused  by  a 
scarcity  of  definitions  for  the  strategies,  and  compounded  by  the  application  of  the  strategies 
in  engineering  disciplines  far  removed  from  where  the  terms  were  first  used.  The  second 
factor  is  a  universal  application  of  the  strategies  to  all  acquisitions,  rather  than  to  just  those 
acquisitions  where  the  strategy  would  be  particularly  relevant  and  helpful. 

The  existing  literature  does  not  provide  sufficient  guidance  on  the  architecting 
strategies,  or  the  types  of  product  acquisitions  where  they  should  be  applied.  The  Defense 
Acquisition  University  (DAU)  provides  an  excellent  start,  with  brief  definitions  of  many  of  the 
important  terms  in  its  Glossary  of  Defense  Acquisition  Acronyms  and  Terms  (Hagan,  2009), 
although  flexibility ,  adaptability,  and  extensibility  are  not  defined.  A  symposium  paper  from 
the  MIT  Engineering  Systems  Division  discussing  architecting  strategies  generally  is  also 
useful,  but  does  not  specifically  define  the  strategies  and  the  differences  between  them 
(Crawley  et  al.,  2004).  The  DoD  dictionary,  which  consolidates  definitions  provided  in 
doctrine  documents,  defines  only  commonality  (Joint  Chiefs  of  Staff,  2010).  Finally,  even 
within  the  top  100  articles  found  doing  a  Google  Scholar  search  for  articles  containing  the 
words  robust,  flexible,  common,  interoperable,  and  extensible,  none  sets  out  definitions  for 
these  terms.  Although  these  searches  may  not  be  exhaustive,  they  represent  a  much  more 
detailed  search  for  definitions  than  a  professional  is  typically  able  to  conduct  when 
investigating  architectural  options. 

The  specific  objectives  of  this  paper,  therefore,  are  twofold.  First,  the  paper  aims  to 
synthesize  existing  definitions  of  the  design  strategies  into  a  single  definition.  This  synthesis 
will  provide  a  starting  point  for  discussion  of  what  the  design  strategies  mean.  Extensive 
comment  and  discussion  about  these  definitions  is  anticipated,  but  consolidating  all 
definitions  into  a  single  document  and  posing  possible  definitions  for  discussion  is  a 
prerequisite  for  this  discussion,  and  represents  an  advance  on  the  existing  literature. 

Second,  the  paper  aims  to  provide  a  coarse  analysis  of  the  acquisition  scenarios  to 
which  each  strategy  is  well  suited.  Such  an  analysis  makes  the  broad  point  that  different 
acquisition  scenarios  merit  different  design  strategies,  and  not  one  design  strategy  is  a 
panacea  for  all  acquisition  challenges.  The  analysis  also  makes  more  specific  findings  about 
the  regions  of  suitability  of  each  design  strategy,  in  terms  of  certainty  of  requirements  and 
openness  to  competition. 

In  Section  I,  the  paper  examines  the  literature  in  detail.  It  presents  a  number  of 
observations  about  the  potential  for  confusion  in  the  existing  literature,  and  also  highlights 


1  Commonality.  “Commonality  is  the  key  to  affordability”  (DoD,  2013).  Interoperability:  “It  is  DOD 
Policy  that .  .  .  Department  of  Defense  pursue  materiel  interoperability  with  allies  and  coalition 
partners”  (Carter,  2009).  Open  Systems  Architectures:  “[Acquisition  Professionals  within  DOD  will  ...] 
require  open  systems  architectures”  (Carter,  2010).  “Program  managers  shall  employ  MOSA 
[Modular  Open  Systems  Approach]  to  design  for  affordable  change,  enable  evolutionary  acquisition, 
and  rapidly  field  affordable  systems  that  are  interoperable  in  the  joint  battle  space”  (DoD,  2008). 
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how  reference  is  frequently  made  to  the  design  strategies  without  an  accompanying 
explanation  of  what  is  meant  by  those  strategies.  In  Section  II,  standard  definitions  for  each 
of  the  chosen  architecting  strategies  (common,  modular,  open,  flexible,  adaptable,  robust, 
extensible,  and  interoperable,  referred  to  as  the  “Eight  Strategies”)  are  proposed.  Each 
definition  is  illustrated  with  examples  from  previous  government  acquisitions.  Section  III 
presents  an  analysis  of  the  newly  defined  design  strategies  against  two  important 
acquisition  parameters:  (1 )  certainty  of  requirements,  and  (2)  number  of  organizations 
involved. 

Finally,  Section  IV  concludes  the  paper  and  presents  suggestions  for  further  work  in 
this  area. 

The  Existing  Literature  Confuses  More  Than  It  Clarifies 

If  architectural  strategies  are  to  be  used  for  complex  government  acquisition  projects, 
there  is  a  need  for  all  involved  to  understand  the  meaning  of  these  strategies.  This  section 
of  the  paper  reviews  the  existing  definitions  of  architectural  strategies  and  considers 
whether  the  definitions  are  consistent  and  easy  to  find.  Definitions  that  are  consistent  and 
easy  to  find  would  be  expected  as  a  prerequisite  to  effectively  using  the  architectural  design 
strategies  across  government  acquisitions. 

At  the  outset,  it  is  important  to  be  precise  with  terminology  used  in  this  paper.  An 
architecture  is  “an  abstract  description  of  the  entities  of  a  system  and  the  relationships 
between  those  entities”  (Crawley  et  al.,  2004).  Put  another  way,  architecture  is  “the 
arrangement  of  the  functional  elements  into  physical  blocks”  (Ulrich  &  Eppinger,  2008). 
Architecture  is  the  underlying  concept  of  how  a  complex  system  is  brought  together,  the 
process  of  relating  form  to  function  in  order  to  create  value  where  value  is  benefit  at  cost. 

Different  architectures  have  different  properties.  When  the  architecture  clearly  brings 
about  a  certain  property  in  the  final  system,  the  final  system  is  often  referred  to  as  having  a 
“property-architecture.”  For  example,  if  the  architecture  is  such  that  the  final  system  is 
robust,  then  the  system  is  described  as  having  a  robust  architecture.  We  refer  to  the  process 
of  designing  an  architecture  for  a  desired  result  as  an  architecting  strategy. 

Surprisingly,  we  were  able  to  find  only  a  small  number  of  references  that  presented 
and  compared  all  or  most  of  the  architecting  strategies.  De  Week,  Ross,  and  Rhodes  (2012) 
investigated  most  of  the  architecting  strategies  presented  in  this  paper  but  were  concerned 
with  them  as  “system  properties”  rather  than  architecting  strategies.  Their  paper  also 
focused  on  the  interrelationships  between  the  strategies  rather  than  describing  the 
strategies  themselves.  The  symposium  paper  by  the  MIT  Engineering  Systems  Division,  The 
Influence  of  Architecture  in  Engineering  Systems  (Crawley  et  al.,  2004),  investigated 
definition  in  more  detail.  The  paper  described  how  architecture  influences  the  properties  of 
created  systems,  resulting  in  robustness,  adaptability,  flexibility,  safety,  and  scalability. 
However,  Crawley  et  al.  focused  on  the  importance  of  architecture  rather  than  detailing 
different  outcomes  from  the  architecting  process.  Fricke  and  Schultz  (2005)  presented  an 
excellent  side-by-side  view  of  adaptability,  agility,  flexibility,  and  robustness,  but  they  did  not 
extend  their  analysis  beyond  these  “changeable”  architectures. 

Finding  other  papers  that  compared  architecting  approaches  proved  difficult.  A 
Google  Scholar  search  for  articles  containing  the  words  robust,  flexible,  common, 
interoperable,  and  extensible  gives  a  surprising  number  of  results — 13,800 — but  no  paper  in 
the  top  100  sets  out  definitions  for  these  terms.  In  other  cases,  papers  defined  a  few  of  the 
terms  in  domain-specific  areas.  For  example,  Ferguson,  Siddiqi,  Lewis,  and  de  Week  (2007) 
examined  flexible  and  reconfigurable  systems  in  product  design,  but  their  definitions  would 
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require  thought  and  interpretation  before  application  to  another  area,  for  example, 
information  architectures. 

Nor  is  there  assistance  from  the  key  textbooks  in  the  area.  The  Art  of  System 
Architecting  mentions  some  of  these  architecting  strategies  but  does  not  contrast  them  or 
provide  extensive  detail.  In  Architecture  and  Principles  of  System  Engineering,  Dickerson 
and  Mavris  (2010)  did  not  present  definitions  for  any  of  the  Eight  Strategies.  However,  the 
terms  themselves  are  mentioned,  in  some  cases,  in  the  sense  we  use  them  (“interoperable 
and  cost  effective  military  systems”  [Dickerson  &  Mavris,  2010,  p.  148],  “methods  and 
techniques  to  ...  design  for  robustness  relative  to  uncertain  operational  environments”  [p. 
313]),  and  in  other  cases  in  very  different  contexts  (for  example,  openness  is  used  in  the 
context  of  stakeholder  discussions,  and  flexibility  in  the  context  of  “development  flexibility, 
such  as  environmental  limitations  or  regulatory  standards”). 

In  the  government  context,  the  situation  does  not  improve  greatly.  A  complete  set  of 
definitions  does  not  exist.  Of  the  Eight  Strategies,  the  DoD  dictionary,  which  consolidates 
definitions  provided  in  doctrine  documents,  defines  only  commonality  (Joint  Chiefs  of  Staff, 
201 0).  The  DAU  provides  brief  definitions  of  many  of  the  important  terms  in  its  Glossary  of 
Defense  Acquisition  Acronyms  and  Terms,  although  flexibility,  adaptability,  and  extensibility 
are  not  defined  (Hagan,  2009).  Further,  some  of  the  terms  are  narrowly  defined.  For 
example,  module  is  used  only  in  the  context  of  software  architectures.  The  DAU’s  online 
“Terms  and  Definitions”  (2013)  defines  three  out  of  the  Eight  Strategies,  and  defines  the 
substance  of  extensibility,  though  referring  to  it  as  scalability. 

Compounding  the  problem,  the  same  term  is  used  in  the  same  community  to  mean 
different  things.  Defense  Directive  5000.01  (Wolfowitz,  2007)  emphasizes  five  key 
acquisition  policies,  one  of  which  is  flexibility.  Dickerson  and  Mavris  (2010)  summarized 
flexibility  in  this  context  as  the  “need  to  structure  each  acquisition  program  according  to  the 
set  of  strategies,  documentation,  reviews,  and  phases  that  make  sense  for  this  program”  (p. 
290).  This  is  a  different  definition  of  flexibility  than  used  by  system  architects  in  describing 
the  properties  of  their  systems.  However,  there  are  some  bright  spots  in  the  government 
landscape.  The  push  towards  a  “modular,  open-systems  architecture”  by  the  Open  Systems 
Joint  Task  Force  (2004),  shows  significant  development  of  the  modular  and  open  systems 
concepts  through  tens  of  pages  of  principles,  definitions,  and  examples. 

Definitional  confusion  is  not  entirely  due  to  a  deficiency  in  the  literature,  however. 

The  architecting  strategies  are  often  mentioned  in  the  same  breath  but  in  fact  are  concerned 
with  quite  dissimilar  things.  Adaptability,  flexibility,  and  robustness  are  characteristics  of  an 
end-product  that  describes  how  the  product  interacts  with  its  environment,  especially  as  that 
environment  changes.  Extensibility  describes  how  the  product  is  able  to  improve  over  time. 
Interoperable  describes  how  the  product  interacts  with  other  products  in  the  operations 
phase.  Commonality  describes  similarities  with  other  products,  usually  in  the  development 
and  operations  phases.  Modularity  describes  the  physical  structure  of  the  product. 

Openness  describes  the  process  of  acquiring  the  product.  Therefore,  it  is  not  surprising  that 
a  single  paper  does  not  cover  the  range  of  architecting  strategies,  because  they  are  quite 
different. 

Further,  the  architectural  strategies  inter-relate.  Modularity  emphasizes  simple,  well- 
understood  interfaces  and  so  enables  commonality  (through  reuse  of  existing  products)  and 
openness  (by  more  easily  tying  together  the  contributions  of  different  participants). 
Interoperable  architectures  require  knowledge  of  the  systems  that  interoperate,  implying 
some  level  of  openness.  Interoperable  architectures  also  work  because  of  some  degree  of 
commonality,  usually  in  the  patterns  of  information  exchange,  so  an  interoperable 
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architecture  could  also  be  described  as  having,  for  example,  a  common  communications 
protocol. 

The  difficulties  with  using  the  existing  literature  to  define  architecting  strategies  can 
therefore  be  summarized  as  follows: 

•  No  single  reference  presents  and  compares  all  the  architecting  strategies. 

•  Several  key  references  refer  to  architecting  strategies  without  defining  them, 
assuming  that  they  are  well  understood. 

•  Where  definitions  are  provided,  they  are  often  domain-specific. 

•  The  words  chosen  for  the  architecting  strategies  are  sometimes  used  by  the 
same  communities,  with  different  meanings. 

•  The  strategies  interrelate,  and  multiple  architecting  strategies  are  often  used 
to  achieve  a  given  result. 

Although  this  may  appear  a  formidable  list  of  obstacles,  clear,  widely  available 
definitions  of  the  architecting  strategies  will  assist  with  resolving  all  of  these  difficulties.  In 
the  defense  acquisition  context,  referring  to  the  definitions  of  these  strategies  when 
proposing  them  as  mandatory  considerations  in  acquisition  would  improve  communication  of 
the  desired  outcomes.  There  is  a  precedent  for  such  definitional  foundation  in  the 
commonality  literature.  The  RAND  Corporation  produced  a  report  containing  a  standard 
commonality  lexicon  (Held,  Lewis,  &  Newsome,  2007).  A  similar  report  examining  the 
definitions  described  in  Section  II,  with  more  detail  and  rigor,  presents  a  possible  solution  to 
the  current  confusion  in  the  literature. 

Defining  Architectural  Strategies 

In  an  attempt  to  remedy  some  of  the  confusion  outlined  in  the  previous  section,  this 
section  provides  an  overview  of  architecting  strategy  definitions  from  the  engineering 
literature,  a  relevant  DoD  example  of  each  definition,  a  discussion  of  the  definition  as  it 
relates  to  process  versus  architecture,  and,  because  these  strategies  are  often  painted  as  a 
panacea  for  all  new-product  development  ills,  a  description  of  the  possible  downsides  of  the 
approach.  In  the  definitions  that  follow,  we  begin  by  discussing  a  simple  example  of  each 
architecture  strategy.  Because  low  complexity  examples  are  rare  in  the  real  world,  we  also 
discuss  the  application  of  each  strategy  to  a  more  complex,  and  where  possible,  “system  of 
systems”  example. 

Flexible  Architectures 

A  flexible  architecture  is  one  that  is  easily  modified  to  respond  to  changing 
requirements  (Crawleyet  al.,  2004;  Fricke  &  Schultz,  2005;  Ferguson  et  al. ,  2007).  The 
modification  requires  work  to  be  done  on  the  system.  For  example,  an  architecture  that 
allows  different  external  stores  (often  referred  to  as  pods)  to  be  loaded  on  military  aircraft  to 
provide  different  functionality  for  different  missions  would  be  described  as  a  flexible 
architecture.  External  stores  can  provide  numerous  functions,  including — but  not  limited  to — 
weapons,  additional  fuel,  electronic  counter  measures  (ECMs),  communications,  and 
sensors.  A  more  complex  example  is  the  ability  to  load  different  software  onto  pre-defined 
hardware,  such  as  is  expected  from  software  programmable  radios.  Loading  new  software 
changes  the  functionality  of  the  radio  to  suit  the  operating  environment.  In  each  case,  the 
designers  considered  that  easily  changing  the  system  performance  was  important,  and 
allowance  for  such  changes  was  built  into  the  architecture.  The  architectural  choices  permit 
product  flexibility  and  therefore  are  described  as  a  flexible  architecture.  The  benefit  of  a 
flexible  architecture  is  that  a  particular  design  continues  to  perform  even  as  the  environment 
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changes.  For  example,  an  entirely  new  airplane  is  not  required  simply  because  the  range 
requirements  for  a  certain  mission  exceed  the  internal  fuel  storage  capacity  of  the  aircraft. 

Flexibility  is  often  associated  with  modularity  if  the  flexibility  arises  through  the 
swapping  of  modular  parts  (for  example,  weapons).2  Flexibility  need  not  be  dependent  on  a 
modular  architecture,  however.  A  flexible  system  could  allow  for  software  changes  to  be 
inserted  without  even  unpacking  the  part  from  the  case,  as  in  the  case  of  the  Block  III  High 
Speed  Anti-Radiation  Missile  (HARM),  a  system  designed  to  destroy  radar  equipped  air 
defenses.  The  Block  II  HARM  has  its  own  software  operating  system,  which  can  be 
upgraded  in  the  field — and  in  the  crate — to  redefine  its  flight  profile,  its  function,  and  how  it 
interacts  with  the  targeting  system  onboard  the  aircraft  system. 

Flexibility  is  not  always  a  positive  attribute,  however,  as  there  is  a  price  associated 
with  designing  systems  to  be  flexible.  Flexibility  should  be  used  only  where  uncertainty  of 
requirements  for  the  system  means  that  the  strategy  is  required.  Crawley  et  al.  (2004)  put 
this  succinctly: 

In  some  cases,  flexibility  comes  at  a  price — namely,  efficiency  in  some  form. 
Flexibility  may  require  over-design,  generic  components,  extra  interfaces,  or 
changeover  time.  A  less  flexible  system  might  have  more  focused 
components,  fewer  interfaces,  and  no  loss  due  to  changeover. 

Flexibility  can  in  fact  increase  overall  lifetime  costs  of  a  system,  especially  if  the 
product  lifetime  is  shorter  than  expected,  due  to  the  significant  up-front  cost.  As  the  Army’s 
Future  Combat  System  program  office  pointed  out  in  its  reaction  to  a  GAO  (2009)  report, 

Because  of  the  significant  amount  of  new  technology  development  and  the 
emphasis  on  laying  a  good,  flexible  architecture  foundation,  development 
effort/costs  may  not  follow  typical  expenditure  rates  as  other  projects,  and  a 
larger  percentage  will  be  needed  in  the  early  stages  of  the  program. 

Adaptable  Architectures 

Fricke  and  Schultz  (2005)  described  adaptable  systems  as  systems  that  “deliver  their 
intended  functionality  under  varying  operating  conditions  through  changing  themselves.”  In 
other  words,  an  adaptable  architecture  modifies  itself  to  meet  a  changing  environment.  An 
example  from  the  commercial  world  is  commercial  power  generation,  which  automatically 
brings  additional  power  production  online  during  high  demand  periods.  In  the  defense 
context,  an  example  of  an  adaptable  system  is  radar.  Most  radar  systems  are  able  to 
change  their  receiver  gain  automatically  in  order  to  filter  out  noise  generated  by  jamming. 

The  difference  between  flexible  and  adaptable  is  subtle,  and  in  the  experience  of  the 
authors,  those  using  the  terms  do  not  always  grasp  the  difference.  In  particular,  either  term 
is  often  used  as  a  catch  all  for  the  meaning  of  both  terms.  The  difference  between  adaptable 
and  flexible  architectures  has  important  cost  implications  for  DoD  projects.  Adaptability 
usually  places  significantly  greater  demands  on  a  system  than  flexibility  but  may  be 
warranted  in  some  cases,  for  example,  where  human  intervention  is  impossible3  (such  as  a 
pacemaker),  or  where  human  reaction  times  are  too  slow  (for  example,  the  ACESII  ejection 
seat,  which  automatically  changes  its  ejection  profile  based  on  the  altitude  and  airspeed  of 
the  aircraft  at  the  time  of  ejection). 


2  De  Week,  Ross,  and  Rhodes  (2012)  showed  this  as  a  strong  link  in  their  diagram  of  “ility  co¬ 
occurrence  in  the  literature.” 

3  In  the  DoD  context,  adaptability  in  the  context  of  situations  where  human  intervention  is  impossible 
is  tied  to  autonomy,  which  is  commonly  not  acceptable  given  the  high  stakes  involved  in  warfare  and 
the  unwillingness  to  take  the  human  decision-maker  out  of  the  loop. 
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There  is  also  overlap  with  other  terminology.  An  open  architecting  process  could  be 
described  as  a  flexible  architecture  because  it  allows  new  design  implementations  to  be 
introduced  over  time.  An  “extensible”  architecture  can  also  be  changed  over  time  and 
therefore  be  characterized  as  “flexible,”  albeit  at  the  most  inflexible  end  of  flexible.4  Finally,  a 
modular  approach  enables  flexibility  but  is  not  enough  in  itself  to  guarantee  flexibility.5  To 
conceptualize  this,  imagine  a  modular  system,  such  as  the  aircraft  with  weapons  discussed 
previously,  where  the  weapons  racks  were  welded  to  the  aircraft  frame.  The  design  is  no 
less  modular,  but  the  architecture  is  no  longer  flexible. 

Robust  Architectures 

A  robust  architecture  is  one  that  is  able  to  meet  its  performance  specification  over  a 
wide  range  of,  often  unanticipated,  external  conditions  and  still  perform  well  (Hagan,  2009; 
Crawley  et  al.,  2004;  Fricke  &  Schultz,  2005).  This  design  strategy  is  often  used  when  there 
is  high  uncertainty  over  the  future  performance  requirements  of  a  product  (Thomke,  1 997), 
or  when  the  system  itself  is  complex  and  not  well  understood  (Crawley  et  al.,  2004).  The 
design  approaches  to  achieve  robustness  are  not  well  understood  (Crawley  et  al.,  2004), 
particularly  in  the  area  of  software  design.  The  benefit  of  a  robust  architecture  is  that  the 
product  keeps  performing  even  as  the  external  environment  changes.  Robustness  may  be 
preferred  to  flexibility  or  adaptability  for  a  number  of  reasons.  Robustness  may  be  a  lower 
cost  approach  because  the  system  never  needs  to  change.  Robustness  may  also  be 
preferred  for  situations  where  the  range  of  environmental  challenges  is  not  well  known.  An 
example  of  a  robust  architecture  from  the  defense  context  is  the  design  of  the  Link-16 
protocol,  which  assumed  that  message  traffic  might  get  lost  in  the  dynamic  airborne 
environment.  Therefore,  it  built  significant  redundancy  into  its  message  traffic,  sending 
positional  data  and  other  messages  multiple  times  per  second  to  ensure  delivery.  A  classic 
example  of  a  robust  architecture  is  the  nuclear  command  and  control  architecture.  Built  into 
mountains  and  underground  silos,  and  designed  to  operate  in  a  post-nuclear  attack  radiation 
environment,  robustness  was  clearly  a  main  design  criteria. 

System  designs  described  as  “robust”  are  more  widely  used  than  the  strict  definition 
above  would  allow.  Some  consider  a  robust  design  to  be  anything  that  copes  with 
environmental  changes  and  continues  to  perform.  Robustness  is  also  used  as  a  synonym 
for  survivability,  to  indicate  continued  performance  when  components  of  the  system  are 
damaged.  Finally,  some  members  of  the  defense  community  (perhaps  showing  some 
pessimism  with  the  acquisitions  process)  use  a  robust  design  to  mean  one  that  actually 
works  as  designed  under  field  conditions,  using  it  interchangeably  with  “ruggedized” 
(Hawkes,  2013;  Sherborne  Sensors,  2013).  To  add  to  the  confusion,  Thomke’s  (1997) 
paper,  which  contains  excellent  case  studies  into  what  we  would  call  “robustness”  in  the 
design  stage,  describes  the  cases  as  “design  flexibility.” 

It  is  obvious  that  a  robust  architecture  will  usually  be  more  expensive  upfront  than  a 
conventional  architecture.  The  greater  span  of  requirements  often  necessitates  more  time 
preparing  better  designs  or  more  cost  in  manufacturing,  as  more  exotic  materials  are  used. 
Therefore,  as  with  any  architectural  design  choice,  there  is  a  cost-benefit  tradeoff  for  a 
robust  design. 


4  For  clarity,  systems  that  are  intended  to  be  changed  back  and  forth  many  times  are  usually  referred 
to  as  “flexible  and  reconfigurable”  (Ferguson  et  al.,  2007). 

5  A  simple  illustration  of  the  link  between  flexibility  and  modularity  can  be  seen  with  a  Google  Scholar 
search  for  (“flexible  and  modular”  or  “modular  and  flexible”),  which  yields  ten  times  more  results  than 
(“flexible  and  robust”  or  “robust  and  flexible”). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  85- 


Open  Architectures 

Open  architectures  are  becoming  increasingly  popular  due  to  their  prevalence,  and 
success,  in  the  software  industry.  Silver  (2010)  examined  the  history  of  open  architectures  in 
detail.  An  open  architecture  is  one  where  the  necessary  information  to  design  a  part  of  the 
system  is  made  accessible  to  the  public  or  a  wide  group  of  possible  designers.  Hagan 
(2009)  goes  into  more  detail,  defining  an  open  system  as 

a  system  that  implements  specifications  maintained  by  an  open,  public 
consensus  process  for  interfaces,  services,  and  support  formats,  to  enable 
properly  engineered  components  to  be  utilized  across  a  wide  range  of 
systems  with  minimal  change,  to  interoperate  with  other  components  on  local 
and  remote  systems,  and  to  interact  with  users  in  a  manner  that  facilitates 
portability. 

The  essence  of  these  definitions  is  that  open  architectures  allow  any  interested 
organizations  to  participate  in  the  design  and  development  of  parts  of  the  system.  This  is  not 
a  new  idea,  but  the  fact  that  individuals  anywhere,  equipped  with  only  a  computer,  can 
contribute  to  open  software  development,  combined  with  the  increased  importance  of 
software  to  complex  projects,  has  meant  that  the  pool  of  potential  contributors  to  open 
architectures  has  widened  over  recent  decades.  The  benefit  of  an  open  architecture  is  that 
better  solutions  can  sometimes  be  found  because  more  organizations  have  the  chance  to 
examine  the  problem  and  propose  design  solutions.  The  increased  competition  also  has  the 
potential  to  lower  costs. 

An  example  from  the  defense  context,  though  not  yet  officially  sanctioned,  is  the 
growth  of  “Tactical  iPhone  apps”  that  have  been  developed  both  by  soldiers  and  by  small 
companies  (Tactical  Nav,  2013).  These  are  built  using  the  open  interface  exposed  to 
applications  developers  by  Apple. 

Openness  is  generally  well  understood  and  difficult  to  confuse  with  any  of  the  other 
terms  presented  here.  It  is  important  to  recognize  that  openness  is  more  concerned  with  the 
process  of  development  than  the  attributes  of  the  end-state  of  the  product.  However,  the 
system  architect  is  concerned  with  process  as  well  as  end-state  because  the  development 
process  affects  affordability  by  changing  development  cost.  Therefore,  in  developing  and 
comparing  architectural  strategies,  it  is  valid  to  consider  strategies  that  affect  process. 

Open  architectures  have  some  significant  drawbacks  that  are  sometimes  overlooked 
in  the  current  enthusiasm  for  their  use.  Open  architectures  present  coordination  challenges 
for  the  government  customer  who  must  ensure  that  the  products  developed  on  the  open 
market  can  interface  to  produce  a  usable  end  product.  The  broad  dissemination  of 
information  about  the  end  product  may  also  present  security  concerns. 

Common  Architectures 

A  common  architecture  focuses  on  reuse  of  proven  systems,  or  the  design  of 
platforms  for  later  reuse  (Wicht,  2011).  With  relatively  simple  systems,  the  key  benefit  is  cost 
reduction,  as  much  of  the  work  from  the  first  system  is  reused.  Reusing  systems  and/or 
system  components  also  decreases  the  development  time  associated  with  the  system.  With 
more  complex  systems,  other  benefits  also  become  obvious:  reliability  increases  because 
proven  designs  are  reused  and  each  part  is  used  more  often;  maintenance  and  logistics  are 
more  affordable  because  there  are  fewer  unique  parts;  less  training  is  required  as  operators 
are  familiar  with  previous  instances  of  the  product. 
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Examples  from  a  DoD  perspective  include  the  Joint  Strike  Fighter  (JSF),  which  was 
designed  in  three  variants  with  as  many  common  parts  as  possible.6  Another  example  is  the 
M61A1  20  mm  cannon.  This  automatic  weapon,  often  called  the  Vulcan,  has  been  used  in 
numerous  Air  Force  aircraft  (F-104,  F-105,  F-106,  F-4,  F-14,  F-15,  F-16,  F-18,  A-7,  F-111D, 
and  most  recently  the  F-22),  the  Navy  PHALANX  system,  and  the  Army  C-RAM. 

Commonality  is  straightforward  in  principle;  however,  it  has  significant  overlap  with 
modularity  and  interoperability.  In  a  modular  system,  multiple  common  modules  are  often 
used  to  incrementally  increase  performance.  The  resulting  system  has  strategies  of  both 
modularity  and  commonality.  For  example,  some  launch  vehicles  have  a  modular 
configuration  with  respect  to  the  number  of  solid  rocket  boosters  clustered  around  the 
vehicle  core.  For  example,  the  Atlas  V  launch  vehicle  can  have  from  zero  to  five  solid  rocket 
boosters  attached  to  the  core,  depending  on  the  particular  payload  and  orbit  of  a  launch.7 

Modularity  also  makes  reuse  easier  and  therefore  enables  commonality  at  a  lower 
cost.  Software  modules  are  the  canonical  example  of  this,  because  good  practice  software 
writing  encapsulates  particular  software  tasks  into  modules,  with  defined  inputs  and  outputs. 
If  that  functionality  is  required  in  a  subsequent  development,  the  module  can  be  easily 
transplanted  into  the  new  context.  Commonality  and  interoperability  are  also  blurred. 
Interoperability  generally  requires  a  common  (or  “standardized”)  interface.  Therefore,  the 
two  systems  are  interoperable,  or  they  share  a  common  interface.  The  outcome  is  the  same, 
but  the  terminology  could  be  used  differently.  We  suggest  that  if  commonality  is  used  solely 
for  standardizing  communications  protocols  for  interoperability,  the  guiding  strategy  is 
interoperability.  However,  if  the  rationale  is  life-cycle  cost  savings  from  common  design  of 
terminals,  hardware,  or  training  procedures,  then  “commonality”  is  probably  more 
appropriate. 

Commonality  does  not  always  produce  benefits.  For  example,  if  requirements 
change  and  a  new  system  is  required,  the  additional  up-front  investment  in  designing  a 
common  system  is  lost.  In  some  instances,  the  cost  of  designing  and  enforcing  a  common 
system  outweighs  the  life-cycle  cost  savings  of  having  the  common  system.  This  may  be  the 
case  with  the  F-35,  which  has  had  “continuing  manufacturing  inefficiencies,  parts  problems, 
and  technical  changes  [that]  indicate  that  the  aircraft’s  design  and  production  processes 
may  lack  the  maturity  needed  to  efficiently  produce  aircraft  at  planned  rates”  (GAO,  2011). 
The  program  was  restructured  in  2011,  triggering  a  Nunn-McCurdy  unit-cost  breach. 
Performance  is  often  penalized  with  a  common  system,  with  both  systems  having  to  share  a 
system  that  suits  neither  of  them  perfectly. 

Modular  Architectures 

To  deal  with  complexity  in  systems,  the  idea  of  modularity  is  as  old  as  engineering 
itself.  Modularity  allows  a  complex  problem  to  be  tackled  in  pieces.  At  its  most  basic,  a 
modular  architecture  focuses  on  dividing  the  form  of  the  system  to  reflect  the  functions  of 
the  system.  This  means  that  the  system  can  be  divided  into  chunks,  each  of  which  performs 
a  distinct  function.  Baldwin  and  Clark  (1999)  had  an  elegant  definition:  “A  module  is  a  unit 
whose  structural  elements  are  powerfully  connected  among  themselves  and  weakly 
connected  to  elements  in  other  units.”  This  design  strategy  tends  to  produce  “tidier”  designs 


6  “The  JSF  program  goals  are  to  develop  and  field  a  family  of  stealthy  strike  fighter  aircraft  for  the 
Navy,  Air  Force,  Marine  Corps,  and  U.S.  allies,  with  maximum  commonality  to  minimize  costs”  (GAO, 
2009). 

7  United  Launch  Alliance  described  the  Atlas  V  under  the  heading  “Modular  System  for  Maximum 
Flexibility  and  Reliability”  as  using  “a  standard  common  core  booster™  (CCB),  up  to  five  strap-on 
solid  rocket  boosters  (SRB)”  (United  Launch  Alliance,  2012). 
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with  associated  benefits  to  reliability  and  re-work  cost.  Modularity  standardizes  interfaces 
and  minimizes  the  amount  of  information  that  needs  to  travel  across  those  interfaces.  More 
advanced  modularity  defines  standard  interfaces  between  aspects  of  the  form  of  the  system, 
and  allows  those  pieces  of  form  to  be  swapped  out  to  produce  different  functions.  This 
allows  the  product  to  perform  across  a  greater  range  of  external  environments  and  to  be 
upgraded  more  quickly  and  cheaply.  For  example,  the  computer  and  USB-peripheral 
architecture  now  used  on  personal  computers  is  a  modular  architecture.  The  defined 
interface  is  the  USB,  and  different  forms  with  different  functions  can  be  connected  to  the 
computer  via  USB  to  improve  the  function  of  the  system  as  a  whole.  A  defense  example  of 
modularity  is  the  guided  bomb  unit  (GBU).  GBUs  are  basically  a  series  of  modular  parts, 
including  guidance  systems,  ordnance,  and  fuses,  among  others,  that  can  be  assembled 
from  the  modules  based  on  the  need.  Depending  on  the  target  and  the  desired  effect, 
weaponeers  basically  build  munitions  from  standard  modular  parts.  There  are  a  number  of 
different  approaches  to  modularity  (Crawley  et  al. ,  2004),  but  all  revolve  around  the  same 
idea  of  neatly  encapsulating  product  functions  inside  aspects  of  form. 

Discussions  about  modularity  usually  imply  that  modularity  is  beneficial  for  product 
development;  however,  this  is  not  always  the  case.  Modularity  is  beneficial  if  it  assists  the 
product  in  meeting  its  cost  and  performance  goals,  for  example  through  enabling 
commonality,  flexibility,  or  simply  neater  design  with  less  re-work.  Modularity  requirements 
can  be  detrimental  in  applications  where  performance,  space,  or  weight  is  at  a  premium.  In 
these  cases,  modularizing  the  system  may  introduce  unacceptable  performance  penalties. 
For  example,  an  iPhone  is  a  tightly  integrated  system.  The  touchscreen  and  camera  are 
built  into  the  casing,  and  the  batteries  are  such  an  integral  part  of  the  unit  that  they  cannot 
be  separately  replaced.  This  allows  the  iPhone  to  be  made  smaller,  but  makes  it  more 
difficult  to  reuse  sections  of  the  phone  from  model  to  model.  Changes  to  the  internal  design 
between  the  iPhone  4  and  the  iPhone  4S  meant  that  the  positions  of  buttons  on  the  case 
needed  to  shift. 

This  tight  interaction,  where  changes  to  one  part  of  the  product  necessitate  other 
changes,  is  typical  of  tightly  integrated  systems.8  A  second  example  is  writing  high- 
performance  software.  The  use  of  “libraries”  (pre-existing  code,  the  software  equivalent  of 
modules)  is  minimized,  and  their  functionality  often  re-written  completely  in  order  to  optimize 
it  for  a  particular  application.  Only  the  code  absolutely  required  for  the  program  to  run  is 
included.9 

Modularity  also  shows  significant  interaction  with  other  architecting  strategies, 
particularly  open  architectures.  This  is  because  openness  usually  outsources  many  of  the 
design  tasks,  reducing  the  ongoing  communication  between  the  system  architects  and  the 
product  design  teams,  and  increasing  the  risk  of  integration  difficulties.  Modularity’s 
emphasis  on  clearly  defined  interfaces  and  each  module  performing  a  single  function 
mitigates  integration  risk,  and  therefore  works  well  with  an  open  architecture.  An  example  of 
modularity  and  openness  working  together  is  the  development  of  apps  for  smartphones.  The 
apps  are  modular  add-ons  to  improve  the  functionality  of  the  phone  and  can  be  developed 
by  anyone  (i.e.,  a  partially  open  architecture).  In  the  defense  context,  modularity  has  been 
combined  with  open  systems,  which  modularity  enables.  The  result  is  “modular  open- 
systems  architectures.”  DoD  Directive  5000.1  states  that  “acquisition  programs  shall  be 
managed  through  the  application  of  a  systems  engineering  approach  that  optimizes  total 


8  See,  for  example,  Giffin  et  al.  (2009),  who  found  less  change  propagation  through  a  system  where 
“the  architecture  of  [the]  system  was  carefully  crafted  to  be  modular  from  the  start.” 

9  “In  structured  software  design,  functionality  and  data  is  arranged  in  software  modules”  (Chakrabati, 
de  Alfaro,  Henzinger,  Jurdzinski,  &  Mang,  2002). 
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system  performance  and  minimizes  total  ownership  costs.  A  modular,  open  systems 
approach  shall  be  employed,  where  feasible”  (Wolfowitz,  2007).  This  is  another  example  of 
constructive  interaction  between  two  architecting  strategies. 

Interoperable  Architectures 

Almost  all  systems  are  interoperable  with  some  other  systems  because  nothing 
works  in  absolute  isolation.  Most  electronic  devices  are  interoperable  with  mains  power; 
most  computers  are  interoperable  with  Internet  servers;  most  vehicles  are  interoperable  with 
the  highway  systems  in  the  countries  for  which  they  were  built.  Exceptions  to  these  rules 
exist,  but  only  in  specialized  applications.  When  we  use  the  word  interoperable  architecture, 
therefore,  it  is  not  to  describe  these  common  situations  where  the  interoperability  is  implicit, 
but  rather  to  describe  systems  where  the  interoperability  is  a  key  requirement  of  the  user. 

Further,  interoperability  is  what  defines  systems  in  the  sense  that  if  there  is  no 
interaction,  there  is  no  system.  If  a  broader  perspective  is  taken,  any  product  that  is 
interoperable  with  another  can  therefore  be  seen  as  simply  two  parts  of  a  single  system.  For 
example,  one  type  of  radio  mounted  in  a  ship  could  be  described  as  interoperable  with 
another  type  of  radio  mounted  in  an  aircraft.  Or,  a  broader  system  could  be  considered  that 
includes  both  radios,  in  which  case  the  interoperability  is  internal  to  the  system.  Therefore, 
simply  depending  on  where  the  boundaries  of  the  system  are  drawn,  an  interoperable 
architecture  can  refer  to  interoperability  with  systems  outside  the  architecture  or 
interoperability  with  systems  internal  to  the  architecture. 

The  first  view  of  “interoperable”  is  used  to  describe  architectures  capable  of 
interfacing  with  specified  systems  external  to  the  architecture  under  consideration,  in  order 
to  improve  its  functionality.  This  is  the  usual  level  of  consideration  of  the  architecture  and  is 
the  substance  of  Hagan’s  (2009)  definition  of  interoperability: 

The  ability  of  systems,  units,  or  forces  to  provide  data,  information,  materiel, 
and  services  to  and  accept  the  same  from  other  systems,  units,  or  forces  and 
to  use  the  data,  information,  materiel,  and  services  so  exchanged  to  enable 
them  to  operate  effectively  together. 

Making  a  system  interoperable  usually  increases  the  usefulness  of  that  architecture. 
For  example,  designing  a  radio  handset  that  can  use  existing  waveforms  increases  the 
number  of  other  radios  with  which  it  can  communicate.  The  ability  to  interoperate  external  to 
the  architecture  under  consideration  permits  wider  communication  than  developing  a  new, 
unique  waveform.  This  would  usually  make  a  more  useful  product  than  developing  a  new 
radio  in  isolation. 

The  second  way  is  a  high-level  view  in  which  the  elements  of  the  architecture  under 
consideration  are  themselves  interoperable.  This  second  view  was  referred  to  as  “intra¬ 
operability”  by  the  Open  Systems  Joint  Task  Force  (2004).  For  example,  in  designing  a 
military  communications  network  like  the  Joint  Tactical  Radio  System  (JTRS),  a  guiding 
principle  was  that  any  radio  on  any  platform  running  the  same  waveform  could 
communicate.  The  JTRS  architecture  could  therefore  be  described  as  an  interoperable 
architecture,  with  the  interoperation  occurring  within  the  system.  To  be  more  specific,  this 
high-level  view  is  often  used  for  systems  with  separate  physical  elements  that  communicate 
information  and  where  interoperation  is  not  essential  to  the  design.  It  would  not  be  common 
to  say  that  a  set  of  radios  designed  for  use  by  groups  of  infantry  was  an  “interoperable 
architecture”  because  radios  that  are  not  interoperable  with  each  other  are  generally 
useless.  However,  in  the  case  of  the  JTRS,  where  radios  on  aircraft  could  interface  with 
radios  on  ships  and  in  the  hands  of  infantry,  this  was  an  unprecedented  degree  of 
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interoperation  that  was  central  to  the  JTRS  project.  Therefore,  describing  the  JTRS  as  an 
“interoperable  architecture”  adds  information  about  the  central  design  strategy. 

Interoperable  architectures  also  present  some  disadvantages.  The  increased  cost 
and  complexity  of  design  involved  in  making  an  architecture  interoperable  should  not  be 
overlooked.  In  particular,  interoperable  architectures  are  difficult  to  test  because  the 
boundaries  of  the  system  under  test  are  often  unclear  or  difficult  to  simulate  in  real-world 
conditions. 

A  common  issue  with  implementing  interoperability  is  different  implementation  of  the 
standard.  An  example  is  the  Link-16  standard  message  set,  which  has  been  implemented 
differently  across  various  systems,  resulting  in  suboptimal  interoperability. 

Extensible  Architectures 

An  extensible  architecture  is  one  that  makes  provision  for  additional  elements  to  be 
added  in  the  future.  In  contrast  to  a  flexible  architecture,  where  the  guiding  strategy  involves 
addition  and  removal  as  needed,  an  extensible  architecture  generally  contemplates 
permanent  additions.10  A  striking  example  of  extensibility  is  the  practice  of  constructing 
future  on-off  ramps  at  the  time  of  construction  of  highway  overpasses.  These  “ramps  to 
nowhere,”  which  extend  only  a  short  distance  out  from  the  main  bridge,  minimize  the  cost 
and  disruption  of  traffic  if  another  road  needs  to  be  connected  to  the  overpass  at  a  future 
time. 


In  a  DoD  context,  an  example  of  an  extensible  architecture  is  the  F-15E  Strike  Eagle, 
which,  when  it  was  built,  was  built  and  architected  to  support  four  radios  but  was  initially 
fitted  only  with  two.  However,  the  space,  the  physical  interface,  and  the  interface  with  the 
Operational  Flight  Program  (the  software)  were  all  developed  and  built  in  at  the  start.  One  of 
the  two  remaining  slots  has  been  subsequently  filled.  The  disadvantages  to  the  extensible 
architecture  are  primarily  the  additional  up-front  expense  and  time  of  building  in  the 
extensibility.  The  extensibility  offers  an  easy  target  for  scope  reduction  under  cost  or 
schedule  pressure.  Extensibility  can  also  be  difficult  to  test  in  complex  systems  because  the 
elements  to  be  extended  are  often  not  created;  therefore,  testing  the  interface  under  realistic 
conditions  is  difficult.  When  the  government,  not  the  contractor,  is  the  ultimate  beneficiary  of 
the  cost  savings  of  a  well-engineered  extensible  solution,  there  is  little  incentive  apart  from 
compliance  testing  to  ensure  that  the  extensibility  is  done  well. 

Extensibility  has  a  relatively  clear  definition.  It  can  be  distinguished  from  flexibility 
through  the  permanence  of  the  extensible  addition.  It  can  be  distinguished  from 
interoperability  because  at  the  time  the  extensible  system  is  created,  the  system  it  will 
interoperate  with  is  not  yet  created.  Note  that  extensibility  is  very  similar  to  scalability,  and 
the  two  are  often  interchanged.  Two  criteria  to  distinguish  the  terms  are  proposed  here 
based  on  our  reading  of  the  nuance  in  usage  between  the  two,  but  these  are  by  no  means 
hard  rules.  First,  extensibility  usually  refers  to  a  bounded  addition,  where  scalability  usually 
refers  to  arbitrarily  large  increases  in  quantity.  For  example,  extensibility  could  be  used  in 
the  context  of  adding  a  second  story  to  a  building  or  an  additional  lane  to  a  freeway. 
Scalability  is  more  commonly  used  in  information  systems  when  unbounded  increases  in 
quantity  are  more  feasible.  For  example,  in  a  computer  network  architecture,  a  scalable 
system  indicates  the  ability  to  add  on  more  nodes  arbitrarily.  Secondly,  scalability  also 


10  No  satisfying  formal  definitions  of  extensibility  could  be  found  in  the  literature,  presumably  because 
the  term  was  widely  used  and  understood.  Wikipedia  states,  without  citation  in  its  entry  on 
extensibility,  that  “in  systems  architecture,  extensibility  means  the  system  is  designed  to  include 
hooks  and  mechanisms  for  expanding/enhancing  the  system  with  anticipated  capabilities  without 
having  to  make  major  changes  to  the  system  infrastructure.” 
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connotes  additions  that  are  similar  or  common  to  what  already  exists,  where  extensibility 
could  include  provision  for  something  different.  For  example,  an  architecture  that  envisaged 
adding  a  garage  to  the  side  of  a  house  might  easily  be  described  as  extensible  but  less 
comfortably  as  scalable.  The  ability  to  duplicate  an  existing  garage  would  be  easier  to 
describe  as  scalable  but  could  probably  also  be  described  as  extensible. 

Summary  of  Engineering  Literature  Definitions 

The  definitions  suggested  previously  are  summarized  in  Table  1,  highlighting  the 
engineering  focus  of  the  design  strategy,  as  well  as  some  of  the  confusing  overlaps  of  the 
terminology  used  to  describe  the  end  result.  Note  that  the  end  goal  is  always  to  deliver  the 
desired  performance  at  required  costs,  and  the  architectural  strategies  should  be  considered 
as  a  range  of  tools  to  achieve  that  end. 
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Table  1.  Architectural  Strategies 


Architectural 

Strategy 

Main  Focus 

Major  Benefits 

Major  Drawbacks 

Common 

Parts,  rather  than 
interfaces 

Increased  life-cycle 
affordability 

Manufacturability 

Reliability 

Higher  upfront  costs 

Sub-optimal  performance 

Modular 

Interfaces  (designed, 
minimized,  and 
standardized) 

One-to-one  mapping  of 
function  to  form 

Leads  to  scalability 

Leads  to  flexibility 

Leads  to  commonality 

Sub-optimal  performance 

Added  weight  (in  some  cases) 

Adaptable 

Changes  itself  based  on 
variations  in  the 
environment 

More  affordable  than 
developing  different 
products 

May  improve  survivability, 
reliability,  or  other 
performance  characteristics 

Requires  well-defined 
requirements 

May  require  over-design, 
generic  components,  extra 
interfaces 

Flexible 

Gets  changed  by 
people  in  reaction  to 
changes  in  environment 

More  affordable  than 
developing  different 
products 

May  improve  survivability, 
reliability,  or  other 
performance  characteristics 

Requires  well-defined 
requirements 

May  require  over-design, 
generic  components,  extra 
interfaces 

Robust 

Continues  to  deliver 
performance  despite 
substantial  variations  in 
environment 

More  affordable  than 
developing  different 
products 

May  improve  survivability, 
reliability,  or  other 
performance  characteristics 

Usually  more  expensive 

Lower  performance 

Interoperable 

Standardizes  interfaces 

Improves  performance 

May  improve  affordability 
through  reuse  of  existing 
network  infrastructure 

Effort  to  correctly  interface  with 
existing  systems 

Perpetuation  of  legacy 
standards 

Open 

Necessary  design 
information  made  public 

Encourages  innovation  that 
may  improve  affordability  or 
performance 

Encourages  competition, 
which  may  improve 
affordability  or  performance 

Loss  of  design  control, 
intellectual  property,  and 
project  influence  by  customer 

Extensible 

Provisions  made  for 
future  permanent 
additions 

Improves  affordability, 
assuming  the  extension  is 
used 

Higher  upfront  costs 

Difficult  to  test  in  development 
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Selecting  and  Aligning  Architectural  Strategies  With  Acquisition  Goals 

Equipped  with  a  better  understanding  of  the  architectural  strategies,  we  return  to  the 
challenge  of  how  the  acquisition  community  can  make  sense  out  of  these  terms  and  best 
apply  an  acquisition  strategy  to  achieve  the  desired  end  state.  The  premise  of  this  section  is 
that  some  acquisitions  are  better  suited  to  some  architectural  strategies. 

Against  the  backdrop  of  several  acquisition  reform  efforts,  including  the  Weapon 
Systems  Reform  Act  of  2009,  the  2008  reissuance  of  DoD  Instruction  5000.02  and  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L])  “Better 
Buying  Power”  memorandum  (Carter,  2010),  understanding  the  interrelation  between 
acquisition  strategies  and  architecting  strategies  becomes  increasingly  critical.  These 
reforms  place  an  increased  emphasis  on  the  systems  engineering  phase,  as  well  as  focus 
on  cost  performance  throughout  a  program’s  life  cycle  (GAO,  2012).  The  Weapon  Systems 
Reform  Act  of  2009,  in  particular,  places  an  emphasis  on  competition  throughout  the 
program  life  cycle  (GAO,  2012).  The  result  of  these  efforts  is  that  more  time  and  money  is 
being  spent  prior  to  system  development  or  production,  and  more  emphasis  is  being  placed 
on  competition  at  all  phases,  to  reduce  cost. 

Each  acquisition  is  unique.  In  attempting  to  give  broad  guidance  to  the  acquisition 
community,  this  paper  focuses  on  two  variables  that  change  how  acquisitions  are  conducted 
and  which  architectural  strategies  may  be  most  appropriate:11  First,  the  degree  to  which 
requirements  and  environment  change  from  the  initial  planning  to  the  field-conditions  of  the 
system;  and  second,  the  number  of  contractors  separately  involved  in  delivering  the  end 
system.  We  consider  a  contractor  separately  involved  in  the  acquisition  if  it  is  directly 
responsible  to  the  government  customer,  rather  than  acting  as  a  subcontractor.  Multiple 
contractors  may  be  introduced  because  the  system  under  consideration  is  too  large  (in 
terms  of  cost  or  complexity)  to  give  to  a  single  company  or  to  increase  competition  in  the 
procurement  process.  Deputy  Secretary  Carter  (2010)  has  already  made  the  point  that  he 
wants  increased  involvement  by  a  larger  number  of  firms  under  the  theory  that  it  lowers 
costs,  increases  buying  flexibility,  increases  the  strength  of  the  industrial  base,  and  leads  to 
company-driven  innovation  (in  support  of  competition).  The  Weapon  Systems  Reform  Act  of 
2009  requires  the  use  of  competitive  prototypes  prior  to  systems  development  to  be  a  part  of 
the  acquisition  strategy  (GAO,  2012). 

These  two  variables  lead  us  to  ask  the  following  two  questions  for  each  of  the  Eight 
Strategies: 

1 .  If  this  architectural  strategy  is  used,  how  flexible  can  the  procurement  be  to 
changes  in  the  anticipated  operating  environment  and/or  requirements? 

2.  If  this  architectural  strategy  is  used,  how  difficult  is  it  to  involve  multiple, 
separate  companies? 

The  results  of  asking  these  questions  are  presented  in  Figure  1. 


11  Of  course,  other  variables  may  also  affect  the  choice  of  the  architecting  strategy,  for  example,  the 
remuneration  structure  of  a  contract  (choosing  from  fixed-fee,  cost-plus,  and  incentive-fee,  among 
others).  The  two  variables  we  chose  are  not  as  well  controlled  by  government  than  many  other 
factors  that  affect  acquisitions;  therefore,  the  architectural  approach  needs  to  be  tailored  to  the 
acquisition  variables,  rather  than  the  acquisition  variables  being  tailored  to  the  architectural  approach. 
For  detailed  examples  on  how  acquisition  variables  could  be  tailored,  assuming  a  commonality 
approach  was  taken  (Wicht,  201 1). 
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Figure  1.  Architecting  Strategies  in  Context 


Figure  1  shows  that,  depending  on  where  across  the  spectrum  a  given  procurement 
falls,  there  are  generally  multiple  architecture  options  that  will  achieve  a  good  result,  but 
there  are  also  architectural  approaches  that  are  not  well  suited.  The  proposed  framework 
described  previously  is  offered  as  a  starting  point  for  identifying  potential  architecting 
strategies  based  on  where  on  the  spectrum  a  given  acquisition  is  likely  to  fall.  The 
architecting  strategy  is  directly  tied  to  cost-benefit  trades  for  the  product,  and  as  a  smart 
buyer  and/or  as  a  systems  architect,  the  government  must  be  aware  of  these  architectural 
considerations.  A  detailed  rationale  for  the  position  of  each  entry  on  the  chart  in  Figure  1 
follows. 


Conventional,  Single-Product  Design  Strategy.  Conventional,  single-product 
design  strategy  describes  a  conventional  single  product,  single  contractor  development 
process  where  the  government  specifies  the  requirements  up  front  and  a  single  contractor 
produces  the  product.  It  has  low  tolerance  to  changes  in  the  initial  requirements  because  the 
contractor  has  no  incentive  to  design  outside  the  requirements  given.  There  are  no  defined 
interfaces  at  the  government-contractor  level,  which  makes  simultaneous  competition 
difficult.  The  intellectual  property  usually  rests  with  the  contractor,  which  makes  competition 
over  time  difficult.  This  is  the  paradigm  that  the  DoD  is  attempting  to  leave,  but  it  has  a  place 
in  acquisition.  For  some  small,  non-complex  procurement,  it  might  be  the  right  strategy. 

Common  Strategy.  Common  design  makes  it  a  little  easier  to  introduce  multiple 
companies,  for  example,  because  a  government  furnished  equipment  (GFE)  process  can  be 
used  across  the  common  elements  of  the  architecture.  One  company  supplies  the 
equipment,  and  another  uses  it  in  the  systems  it  is  developing.  However,  the  common 
design  is  “locked-in,”  making  it  very  intolerant  to  changes  in  requirements.  Any  changes 
need  to  be  cascaded  through  two  contract  mechanisms,  between  the  government  and  the 
GFE  supplier,  and  the  contractor  building  the  current  system.  This  increases  time  and  cost. 

Interoperable  Strategy.  The  interoperable  architecture  strategy  is  intended  to  allow 
multiple  different  products  to  interoperate.  Therefore,  it  is  helpful  for  lowering  barriers  to 
involving  multiple  companies.  However,  the  interoperable  standard  needs  to  be  defined  at 
the  outset  because  it  defines  what  aspects  of  the  system  must  be  the  same  in  order  to  have 
interoperability.  The  standard  is  effectively  common  and  brings  the  inflexibility,  which  is  both 
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the  strength  and  the  limitation  of  a  standard.  The  standard  is  very  difficult  to  update  as 
requirements  change,  and  systems  usually  ignore  the  standard  and  break  the  chain  of 
interoperability  in  cases  of  significant  requirements  change.  Note  that  changing  the  standard 
is  not  the  same  as  changing  other  aspects  of  the  elements  that  interoperate.  For  example, 
an  aircraft  may  be  upgraded  to  fly  further  in  response  to  evolving  requirements,  but  so  long 
as  the  communication  system  remains  unchanged,  the  interoperability  will  remain. 

Robust,  Flexible,  or  Adaptable  Strategy.  These  design  strategies  evolved  to  allow 
systems  to  meet  changing  requirements,  even  if  the  requirements  are  not  known  at  the  time 
they  are  developed.  Therefore,  the  strategies  score  high  on  the  changing  requirements  axis. 
However,  the  principles  that  are  used  to  evaluate  robust,  flexible,  or  adaptable  approaches 
must  be  applied  to  the  system  as  a  whole,  using  rigorous  system  engineering  techniques 
across  the  entire  end  product.  This  makes  it  difficult  to  fragment  the  system  and  use  multiple 
companies. 

Extensible  Strategy.  An  extensible  architecture  builds  in  allowances  for  changes  in 
requirements.  However,  the  changes  need  to  be  anticipated  at  the  outset  in  a  way  that,  for 
example,  a  flexible  architecture  does  not.  It  is  difficult  to  build  an  extensible  architecture 
without  an  idea  of  what  will  be  extended.  However,  building  a  flexible  architecture,  such  as  a 
software-defined  radio,  allows  decisions  to  be  made  about  the  changes  once  the  new 
requirements  are  better  known,  for  example,  writing  new  software.  Whether  an  architecture 
is  extensible  does  not  appear  to  have  a  significant  effect  on  the  involvement  of  different 
companies  in  the  development  of  the  architecture.  Arguably,  it  makes  it  slightly  easier  to 
include  additional  companies  if  the  extension  can  be  “re-competed.”  However,  in  many 
cases,  the  degree  of  knowledge  of  the  original  contractor  about  the  system  makes  it  difficult 
for  new  contractors  to  be  competitive. 

Modular  Strategy.  A  modular  architecture  minimizes  the  interfaces  between  parts  of 
a  product  or  system  and  groups  functional  areas  together.  Therefore,  a  modular  architecture 
is  more  suitable  for  the  involvement  of  multiple  companies  because  of  the  ease  of 
partitioning  work  packages.  A  modular  architecture  also  allows  aspects  of  the  architecture  to 
be  changed  out,  if  necessary,  without  redesigning  the  whole  system,  which  makes  it 
reasonably  tolerant  to  changes  in  requirements. 

Open  Strategy.  An  open  architecture  has  low  barriers  to  involving  multiple 
companies.  There  are  fewer  intellectual  property  barriers,  and  companies  are  free  to  submit 
bids  for  pieces  of  work.  An  open  architecture  is  ideally  changed  quickly  as  requirements 
change  because  there  is  a  short  development  cycle  due  to  competition  and  a  minimum  of 
formal  requirements.  It  should  be  noted  that  open  architectures  are  heavily  dependent  on 
agreed  standards  to  manage  the  interfaces  between  the  open  development  and  other  parts 
of  the  system.  If  the  requirement  changes  necessitate  changes  in  the  interfaces  and 
standards,  then  the  benefits  of  openness  to  dealing  with  the  requirements  change  are  lost. 

The  previous  analysis  suggests  that  architecting  strategies  that  are  chosen  largely 
on  “hard  engineering”  concerns  actually  have  implications  for  the  cost  and  other 
programmatics  of  the  project,  and  the  architecting  community  needs  to  start  coming  to  grips 
with  which  architectures  are  most  useful  in  which  situations.  No  one  acquisitions  approach 
can  be  universally  applied  to  all  architecting  strategies.  The  architecting  strategies  suit 
different  acquisition  scenarios,  and  therefore,  much  thought  should  go  into  which  type  of 
architecting  strategy  is  appropriate  for  each  acquisition.  However,  due  to  the  overlap  of 
some  architecting  strategies,  more  than  one  strategy  may  be  successful  for  a  given 
acquisition.  Figure  1  highlights  where  those  architectures  are  more  likely  to  be  successful 
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choices.  This  underscores  Maier  and  Rechtin’s  (2009)  central  thesis  that  “engineering  is 
more  of  a  science,  and  architecting  is  more  of  an  art.” 

Summary 

The  terms  we  have  called  architecting  strategies  in  this  paper — commonality, 
interoperability,  modularity,  flexibility,  extensibility,  robustness,  adaptability,  and  modularity — 
have  all  been  used  at  various  times  as  preferred  solutions  for  reducing  the  cost  and 
schedule  of  government  acquisitions  of  complex  systems. 

There  has  not  been  a  wide  and  consistent  understanding  of  the  full  meaning  of  these 
terms  across  the  acquisition  community.  In  order  to  use  these  terms  to  communicate 
approaches  and  strategies,  all  personnel  involved  must  share  a  common  understanding  of 
the  terminology.  Sections  I  and  II  of  this  paper  attempted  a  first  step  in  this  direction  by 
surveying  the  literature  and  engineering  practice  to  arrive  at  definitions,  strengths,  and 
weaknesses  for  the  architecting  strategies.  Even  with  a  common  understanding  of  the 
strategies,  a  second  danger  presents  itself.  That  danger  lies  in  a  belief  that  particular 
architecting  strategies  are  the  solution  for  all  acquisitions.  In  fact,  as  Section  III  of  this  paper 
showed,  some  architecting  strategies  are  better  suited  to  particular  acquisition  scenarios 
than  others.  Understanding  the  interconnections  between  the  architecting  strategies  and 
acquisition  scenarios  is  essential  to  making  the  right  decisions  at  project  initiation.  The 
importance  of  getting  the  architecting  strategy  right,  through  good  communication  of  ideas 
and  solid  understanding  of  these  interconnections,  cannot  be  overemphasized.  As  Robert 
Spinrad  said,  “In  architecting  ...  all  the  serious  mistakes  are  made  on  the  first  day”  (Maier  & 
Rechtin,  2009).  Spinrad  was  talking  about  software,  but  the  apothegm  applies  equally  to 
other  forms  of  complex  systems.  Better  communication  and  understanding  of  terminology 
cannot  eliminate  mistakes  altogether,  but  they  represent  a  good  first  step. 
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Abstract 

In  the  face  of  both  declining  budgets  and  growing  interoperability  requirements,  the  military 
increasingly  wants  to  consolidate  multiple  needs  into  single  systems  to  be  developed  jointly. 
Unfortunately,  the  track  record  for  joint  system  acquisition  programs  is  mixed,  and  programs 
often  follow  a  familiar  downward  spiral: 

The  stakeholder  programs  that  depend  on  a  joint  system  may  be  skeptical,  fearing 
the  needed  capability  will  neither  meet  their  needs,  nor  be  delivered  as  promised. 
Stakeholders  pressure  the  Joint  Program  Office  (JPO)  to  accommodate  individual 
requirements,  and  the  JPO  may  reluctantly  agree,  driving  up  cost,  schedule, 
complexity,  and  risk — thus  realizing  the  stakeholders’  worst  fears.  These 
performance  issues  encourage  stakeholders  to  leave  the  joint  program,  potentially 
rendering  it  both  operationally  unattractive  and  financially  infeasible. 

This  exemplifies  a  classic  social  dilemma  called  the  “Tragedy  of  the  Commons.”  Much  work 
has  been  done  on  mitigating  social  dilemmas,  but  a  solution’s  success  depends  on  its 
context.  This  paper  describes  the  modeling  of  organizational  decision-making  in  a  joint 
acquisition  program  using  system  dynamics.  This  permits  future  work  to  analyze  the 
effectiveness  of  different  social  dilemma  mitigations  within  the  context  of  joint  programs  by 
using  system  dynamics. 


Introduction 

The  failure  of  acquisition  programs  to  deliver  high-quality  systems  within  cost  and 
schedule  constraints  (GAO,  2005) — especially  those  developing  software-reliant  systems — 
is  all  too  common  in  modern  government  acquisition.  These  recurring  failures  have  a  direct 
adverse  impact  on  the  ability  of  the  Department  of  Defense  (DoD)  to  be  able  to  support  the 
warfighter  with  the  systems  they  need.  Delayed  systems  withhold  needed  capability,  and 
wasted  resources  drain  budgets  that  could  be  used  to  develop  other  systems. 

The  Software  Engineering  Institute  (SEI)  has  a  unique  insight  into  these  failures  from 
regularly  conducting  Independent  Technical  Assessments  (ITAs)  on  specific  programs  to 
determine  why  they  are  experiencing  difficulties.  These  investigations  have  provided 
visibility  into  the  processes  and  forces  at  work  within  these  programs  and  have  produced  an 
understanding  of  the  most  common  ways  that  programs  come  to  face  serious  challenges. 
Acquisition  programs  do  not  fail  solely  for  technical  reasons.  Organizational,  management, 
and  cultural  issues  are  an  additional  set  of  significant  reasons  why  acquisition  programs 
may  substantially  exceed  budget,  overrun  schedule,  deliver  inadequate  quality,  and 
ultimately  even  fail  (Frangos,  1998;  Madachy,  2008). 

This  paper  describes  research  that  is  being  conducted  to  better  understand  the  joint 
acquisition  program  dilemma  and  to  investigate  approaches  to  mitigate  associated 
problems.  The  general  approach  is  to  use  a  causal  loop  diagram  (OLD)  as  a  means  to 
capture  a  current  understanding  of  the  problem  based  on  past  experience  in  both  consulting 
on  joint  programs  and  in  conducting  ITAs.  The  CLD  embodies  an  evolving  theory  of  the  joint 
acquisition  dilemma  that  is  updated  and  refined  through  a  series  of  workshops  held  with  joint 
program  domain  experts  and  decision-makers.  The  evolving  theory  is  further  explored  by 
developing  the  CLD  into  a  fully  executable  system  dynamics  model.  Data  collected  during 
workshops  help  to  guide,  correct,  and  validate  important  aspects  of  the  model.  When  the 
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model  adequately  captures  the  joint  program  dilemma,  it  can  be  used  to  investigate 
mitigations  to  the  problem  through  additional  modeling  of  different  mitigation  approaches. 
Ultimately,  the  most  promising  mitigations  can  be  evaluated  in  the  workshop  context  and 
potentially  in  pilot  tests  during  the  execution  of  actual  joint  acquisitions. 

The  subsequent  portions  of  this  paper  describe  the  progress  that  has  been  made  in 
conducting  this  research.  The  section  Social  Dilemmas  and  Joint  Programs  describes  the 
typical  flow  of  joint  acquisition  program  events.  The  section  System  Dynamics  Background 
provides  an  introduction  to  the  system  dynamics  modeling  approach.  The  section  Workshop 
with  Domain  Experts  describes  the  workshops  that  have  been  held  thus  far,  and  the  primary 
insights  gained.  The  section  The  Joint  Program  Simulation  Model  describes  the  current 
state  of  a  system  dynamics  simulation  model  refined  based  on  feedback  provided  during 
these  workshops.  Key  behaviors  exhibited  by  the  model  support  the  hypothesis  that  joint 
programs  suffer  from  the  “Tragedy  of  the  Commons”  social  dilemma  and  that  joint  program 
participants  may  get  caught  in  a  trap  that  can  lead  to  the  demise  of  the  program.  The 
section  Mitigations  for  the  Joint  Program  Dilemma  describes  the  space  of  potential 
mitigations  and  solutions  to  the  problems  illustrated.  Finally,  the  paper  concludes  with  a 
discussion  of  the  implications  of  this  work  and  some  future  opportunities. 

Joint  Programs 

The  category  of  programs  known  as  “joint”  programs  constitute  a  special  case  within 
DoD  acquisition.  Such  programs  intend  to  provide  a  system,  subsystem,  or  capability  that 
will  fulfill  needs  of,  and  be  funded  or  managed  by,  more  than  one  DoD  service  or 
component.  Joint  programs  are  appealing  because  they  offer  at  least  two  significant 
potential  benefits:  (1)  reducing  costs  by  developing  one  system  as  opposed  to  several 
differing  ones  and  (2)  improving  interoperability  by  providing  a  single  system  or  capability 
that  can  be  used  for  multiple  purposes  in  multiple  contexts.  Joint  programs  are  recognized 
as  being  difficult  to  manage  because  they  have  multiple  stakeholder  programs  intending  to 
use  the  joint  capability  (often  with  differing  needs),  they  may  be  larger  in  size  than  other 
programs,  they  may  be  more  complex  organizationally,  and  they  may  be  geographically 
dispersed — all  causing  increased  levels  of  coordination,  communication,  and  negotiation 
overhead.  At  the  same  time,  joint  programs  are  becoming  increasingly  important  to  the 
military  as  the  need  for  interoperability  grows  and  as  there  is  greater  pressure  on  the  overall 
defense  budget  to  reduce  costs. 

Although  the  focus  of  most  acquisition  programs  is  on  the  complex  system  being 
developed,  it  may  be  overlooked  that  acquisition  programs  themselves,  especially  joint 
programs,  are  complex,  dynamic  systems — and  as  such  can  display  unpredictable  and  even 
seemingly  chaotic  behavior.  This  results  from  the  presence  of  feedback  between  the 
autonomous  actors  populating  different  groups  within  the  acquisition  organization.  Feedback 
in  the  system  produces  non-linear  behavior,  where  changes  in  the  system’s  outputs  may  no 
longer  be  proportional  to  changes  to  the  inputs.  The  complexity  of  this  feedback,  inherent  in 
any  system  involving  interacting  human  beings,  coupled  with  time  delays  between  inputs 
and  outputs  that  obscure  the  relationships  between  cause  and  effect,  can  produce 
unexpected  behavior  in  even  simple  systems.  Such  systems  must  be  analyzed  as  a  whole 
in  order  to  understand  their  behavior,  because  the  problematic  behaviors  often  emerge 
directly  as  a  result  of  these  interactions — and  vanish  when  the  system  is  decomposed  into 
its  component  pieces  for  study. 

Misaligned  Incentives  in  Acquisition 

It  has  been  concluded  in  studies  (Kadish,  2006;  Pennock,  2008)  that  the  incentives 
at  work  in  acquisition  policy  and  governance  are  often  misaligned.  These  misalignments  can 
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cause  a  disconnect  between  the  desired  outcome  and  the  most  promising  ways  of  achieving 
that  outcome.  The  result  of  misaligned  incentives  can  be  shortsighted  acquisition  decision¬ 
making,  potentially  putting  short-term  interests  ahead  of  longer  term  interests,  or  individual 
and  program  interests  ahead  of  PEO  and  service  interests,  thus  turning  planned  cooperation 
into  opposition. 

Many  of  the  misaligned  incentives  seen  in  acquisition  belong  to  a  category  of 
problems  known  as  social  dilemmas.  Social  dilemmas  are  ubiquitous  across  human 
organizations.  They  describe  situations  in  which  the  incentives  align  to  promote  a  solution 
by  the  actors  involved  that  may  be  locally  optimal  but  will  be  suboptimal  at  a  more  global 
level. 


One  common  type  of  social  dilemma  is  called  a  “social  trap”  (Cross  &  Guyer,  1980; 
Kollock,  1998).  In  a  group  context,  a  social  trap  means  that  an  individual  desires  a  benefit  to 
himself  that  will  cost  everyone  else — but  if  all  in  the  group  succumb  to  the  same  temptation, 
then  everyone  is  worse  off.  A  social  trap  is  often  referred  to  colloquially  as  a  “Tragedy  of  the 
Commons”2  (Hardin,  1968).  What  is  noteworthy  about  this  dilemma  is  that  there  is  no  intent 
to  destroy  the  common  resource — it’s  the  combined  actions  of  all  acting  in  their  own  self- 
interest  that  lead  to  the  tragic  result. 

Social  dilemmas  come  in  many  different  forms,  with  many  different  properties,  which 
helps  to  make  them  both  difficult  to  recognize  and  difficult  to  fix.  The  next  section  outlines 
social  dilemmas  in  the  context  of  a  joint  program. 

Social  Dilemmas  and  Joint  Programs 

Joint  programs  are  noted  for  the  unique  challenges  that  they  face  organizationally 
(Lindsay,  2006),  due  in  part  to  the  tension  between  the  individual  programs  and  services 
needing  to  look  out  for  their  own  interests  and  the  Goldwater-Nichols  Department  of 
Defense  Reorganization  Act  of  1986  that  stresses  the  importance  of  all  service  branches 
working  together  both  effectively  and  efficiently.  Because  of  this  seeming  paradox,  there  is  a 
fundamental  social  dilemma  at  the  heart  of  every  joint  program  that  can  be  seen  in  the 
following  narrative,  which  summarizes  the  experiences  of  a  number  of  joint  and  joint-style 
programs  that  the  SEI  has  worked  with: 

A  joint  program  has  six  stakeholder  programs  all  planning  to  integrate  the 
joint  infrastructure  software  that  is  being  developed  to  meet  a  common 
baseline  set  of  requirements.  However,  each  stakeholder  program  then  also 
requests  that  one  or  more  significant  new  requirements  be  added  to  satisfy 
some  custom  needs  of  that  specific  stakeholder  program.  Although  reluctant, 
the  joint  program  manager  agrees  to  the  new  requirements  out  of  fear  of 
losing  stakeholder  programs,  who  might  leave  the  joint  program  to  build  their 
own  custom  software.  As  development  proceeds,  the  additional  requirements 
and  their  resulting  design  changes  and  incremental  development  significantly 
increase  the  total  cost,  schedule,  complexity,  and  risk  of  the  joint 
development  effort.  As  the  schedule  begins  to  slip,  one  stakeholder  program 
realizes  that  the  joint  program  has  put  the  stakeholder  in  danger  of  missing  its 
own  schedule,  and  so  it  leaves  the  joint  program  to  develop  its  own  software. 


2  The  original  story  of  the  “Tragedy  of  the  Commons”  from  the  1 9th  century  envisions  a  group  of 
herders  sharing  an  area  of  grazing  land  called  a  commons.  If  one  herder  decides  to  graze  an  extra 
animal,  then  that  herder  receives  more  benefit  from  the  commons  than  the  others,  and  at  no 
additional  cost  to  himself.  However,  if  all  of  the  herders  follow  suit  and  add  more  animals  according  to 
the  same  reasoning,  they  eventually  reach  the  point  where  the  grass  is  eaten  faster  than  it  can  grow, 
the  cattle  begin  to  starve,  and  ultimately  all  of  the  herders  lose  their  livelihood. 
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Although  one  stakeholder  program  has  left  the  joint  program,  the  incremental 
cost  of  the  more  complex  architecture  that  was  designed  to  support  the 
stakeholder’s  desired  capability  cannot  be  recouped.  The  schedule  delays 
from  the  increased  complexity  and  risk  impact  the  remaining  stakeholder 
programs  as  well,  and  soon  another  stakeholder  program  chooses  to  leave 
the  joint  program.  Exacerbated  by  the  effort  spent  in  re-planning  the  joint 
effort  each  time  a  stakeholder  program  leaves,  costs  continue  to  escalate, 
and  the  development  schedule  lengthens.  The  remaining  stakeholder 
programs  begin  to  reconsider  their  participation  in  the  joint  program,  and 
ultimately  participation  unravels  and  collapses. 

With  this  narrative  in  mind,  a  joint  program  can  be  viewed  as  a  “Tragedy  of  the 
Commons”  in  which  the  commons  is  the  development  resource  of  the  joint  program  office 
and  the  contractor.  The  entire  program  and  the  stakeholder  programs  are  collectively  worse 
off  if  the  stakeholder  programs  choose  to  exploit  the  development  resource  for  their 
individual  gain  by  insisting  on  having  custom  requirements  developed. 

It  is  important  to  note  that  a  “Tragedy  of  the  Commons”  situation  does  not  always 
occur  in  a  joint  program.  It  may  be  the  case  that  strong  leadership  from  the  joint  program 
manager,  or  a  highly  cooperative  culture  within  the  program,  will  prevent  it  from  happening. 
However,  given  the  fact  that  the  incentives  align  to  favor  unilateral  action  by  the  stakeholder 
programs  and  their  services,  unless  specific  preventative  steps  are  taken,  preventing  this 
social  trap  is  more  likely  to  be  the  exception  rather  than  the  rule. 

The  next  section  provides  context  for  the  creation  of  a  system  dynamics  model  of 
this  behavior. 

System  Dynamics  Background 

The  system  dynamics  method  helps  analysts  model  and  analyze  critical  behavior  as 
it  evolves  over  time  within  complex  socio-technical  domains.  A  key  tenet  of  this  method  is 
that  the  dynamic  complexity  of  critical  behavior  can  be  captured  by  the  underlying  feedback 
structure  of  that  behavior.  The  boundaries  of  a  system  dynamics  model  are  drawn  so  that  all 
of  the  enterprise  elements  necessary  to  generate  and  understand  problematic  behavior  are 
contained  within  them.  The  method  has  a  long  history,  as  described  in  Sterman  (2000)  and 
Meadows  (2008). 

System  dynamics  and  the  related  area  of  systems  thinking  encourage  the  inclusion 
of  “soft”  factors  in  the  model  such  as  policy,  procedural,  administrative,  and  cultural  aspects. 
The  exclusion  of  soft  factors  in  other  modeling  techniques  effectively  treats  their  influence  as 
negligible,  which  is  often  an  inappropriate  assumption.  This  holistic  modeling  perspective 
helps  identify  mitigations  to  problematic  behaviors  that  are  often  overlooked  by  other 
approaches. 

Figure  1  summarizes  the  notation  used  by  system  dynamics  modeling.  The  primary 
elements  are  variables  of  interest,  stocks  (which  represent  collection  points  of  resources), 
and  flows  (which  represent  the  transition  of  resources  between  stocks).  Signed  arrows 
represent  causal  relationships,  where  the  sign  indicates  how  the  variable  at  the  arrow’s 
source  influences  the  variable  at  the  arrow’s  target.  A  positive  (S)  influence  indicates  that 
the  values  of  the  variables  move  in  the  same  direction,  whereas  a  negative  (O)  influence 
indicates  that  they  move  in  opposite  directions.  A  connected  group  of  variables,  stocks,  and 
flows  can  create  a  path  that  is  referred  to  as  a  feedback  loop.  There  are  two  types  of 
feedback  loops:  balancing  and  reinforcing.  The  type  of  feedback  loop  is  determined  by 
counting  the  number  of  negative  influences  along  the  path  of  the  loop.  An  odd  number  of 
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negative  influences  indicates  a  balancing  loop;  an  even  (or  zero)  number  of  negative 
influences  indicates  a  reinforcing  loop. 

Significant  feedback  loops  identified  within  the  model  described  here  are  indicated  by 
a  loop  symbol  and  a  loop  name  in  italics.  Balancing  loops — indicated  with  the  label  B 
followed  by  an  identifying  number  in  the  loop  symbol — describe  aspects  of  the  system  that 
oppose  change,  seeking  to  drive  variables  to  some  equilibrium  goal  state.  Balancing  loops 
often  represent  actions  that  an  organization  takes  to  manage,  or  mitigate  a  problem. 
Reinforcing  loops — indicated  with  a  label  R  followed  by  a  number  in  the  loop  symbol — 
describe  system  aspects  that  tend  to  drive  variable  values  consistently  either  upward  or 
downward.  Reinforcing  loops  often  represent  the  escalation  of  problems  but  may  include 
problem  mitigation  behaviors. 


V.ui.ible -anything  of  interest  in  the  problem  being 
modeled. 

G Ik> st  Variable  -  variable  acting  as  a  placeholder 
for  a  variable  occurring  som  ewhere  else 

Positive  Influence  -  values  of  variables  move  in 
the  same  direction  (e  g.,  source  increases,  target 
increases) 

Negative  Influence  -  values  of  variables  move  in 
the  opposite  direction  (e  g.,  source  increases,  the 
target  decreases) 

Delay -significant  delay  from  v\hen  Vferl  changes  to 
when  Var2  changes 

Balancing  Loop  -  a  feedback  loop  that  moves 
variable  values  to  a  goal  state;  loop  color  identifies 
circular  influence  path 

Reinforcing  Loop  -  a  feedback  loop  that  moves 
variable  values  consistently  upward  or  downward; 
loop  color  identifies  circular  influence  path 

Stock  -  special  variable  representing  a  pool  of 
materials,  money,  people,  or  other  resources. 

Flow  -  special  variable  representing  a 
process  that  directly  adds  to  or  subtracts  from 
a  stock . 

Cloud  -  source  or  sink  (represents  a  stock 
outside  the  model  boundary) 


Figure  1.  System  Dynamics  Notation 

The  next  section  discusses  how  the  system  dynamics  modeling  process  was  used  to 
elicit  a  detailed  understanding  of  joint  program  behavior  from  subject  matter  experts. 

Workshop  With  Domain  Experts 

A  series  of  problem  elaboration  workshops-  is  being  used  as  the  primary  method  for 
gaining  feedback  from  acquisition  subject  matter  experts  on  the  current  system  dynamics 
model,  and  for  eliciting  suggestions  for  additional  potential  improvements.  To  date,  a 
shortened  pilot  version  of  the  problem  elaboration  workshop  has  been  conducted  with 
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3  These  workshops  are  covered  by  the  Carnegie  Mellon  University  (CMU)  human  subject  research 
policy,  and  protocol  HS12-237  for  conducting  these  workshops  has  been  approved  by  the  CMU 
Institutional  Review  Board  (IRB).  Nothing  discussed  at  the  workshops  is  tied  to  a  specific  individual  or 
organization. 
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internal  SEI  acquisition  experts  as  well  as  a  full  two-day  workshop  with  program  office  and 
contractor  personnel  from  a  single  joint  program. 

The  problem  elaboration  workshops  are  intended  to  consist  of  personnel  drawn  from 
a  single  joint  program.  Ideally  each  workshop  will  include  a  mix  of  program  management 
and  technical  personnel  as  well  as  personnel  from  both  the  acquirer  and  developer  side. 

The  workshops  last  approximately  two  days  in  order  to  cover  a  substantial  portion  of  the 
relevant  material.  The  top-level  causal  loop  diagram  of  the  dynamic  is  reviewed  as  a  high- 
level  abstraction  of  the  model  because  reviewing  the  entire  system  dynamics  model  is  not 
feasible  for  the  acquisition  subject  matter  experts. 

There  are  two  primary  goals  for  each  problem  elaboration  workshop:  (1)  discuss  the 
current  top-level  loops  in  the  model  causal  loop  diagram  and  have  the  participants  rate  the 
importance  and  accuracy  of  each  loop  using  a  Likert  scale  and  (2)  gain  insight  from  the 
participants  on  any  loops/interactions  that  may  have  been  overlooked.  The  initial  workshop 
focused  on  a  joint  program  designed  to  provide  a  joint  communication  capability  needed  by 
several  services  that  was  to  be  deployed  on  a  number  of  different  platforms  to  allow  for 
effective  communication  between  platforms  belonging  to  multiple  services.  The  participants 
included  personnel  who  had  worked  at  the  government  program  office  and  personnel  from 
the  prime  contractor.  The  workshops  were  effective  in  achieving  their  goals,  and  some  of  the 
results  are  summarized  as  follows. 

Goal  1:  Rating  the  top-level  loops.  After  presentation  and  discussion  of  all  of  the  top- 
level  loops  in  the  CLD  of  the  large  model  (see  Table  1  in  Appendix  A  for  high-level 
descriptions  of  those  loops  and  Figure  1 1  in  Appendix  B  for  a  graphical  depiction),  ratings 
were  obtained  from  all  participants.  Nine  of  the  12  loops  in  the  CLD  (75%)  were  rated  above 
moderately  important.  In  seven  (i.e.,  58%)  of  the  loops,  the  average  accuracy  score  was 
rated  above  moderately  accurate.  Of  these  seven  loops  rated  above  moderately  accurate, 
four  of  these  loops  (33%  of  the  original  12)  were  rated  above  very  accurate.  For  all  12  loops, 
at  least  one  of  the  four  participants  rated  themselves  as  extremely  experienced  in  this  area, 
and  all  loops  had  at  least  two  participants  who  rated  themselves  as  very  or  extremely 
experienced.  Based  on  the  feedback  from  the  participants,  one  section  of  the  CLD  that 
scored  lower  in  importance  was  modified  in  order  to  change  how  stakeholder  programs  may 
influence  others  to  defect,  or  leave  the  joint  program. 

Goal  2:  Overlooked  loops/interactions.  The  workshop  participants  discussed  nine 
additional  interactions  that  they  thought  had  been  important  on  their  joint  program.  The  top 
area  they  thought  should  be  added  addressed  launching  the  program  properly.  The  model 
was  modified  to  address  this  area,  and  additional  ways  of  implementing  this  concept  are 
being  explored.  A  second  area  that  was  identified  as  needing  to  be  addressed  is  the  level  of 
capability  of  the  government  staff,  and  this  has  been  added  to  the  model  as  well. 

Feedback  from  actual  program  personnel  is  critical  to  ensuring  that  the  model 
includes  the  most  important  top-level  interactions.  It  is  also  critical  to  tuning  the  model 
parameters  to  best  simulate  the  performance  of  joint  programs.  Additional  problem 
elaboration  workshops  are  planned  for  the  near  future  to  continue  to  refine  the  model. 

The  Joint  Program  Simulation  Model 

As  described  previously,  the  problem  elaboration  workshop  attendees  were 
presented  with  a  CLD  that  already  described  many  aspects  of  joint  program  behavior.  The 
feedback  from  these  domain  experts  made  it  possible  to  assess  the  most  important  aspects 
of  the  joint  program  problem,  many  of  which  were  included  in  the  original  CLD  and  some  of 
which  were  not.  This  information  was  used  to  develop  a  simpler  and  more  focused  CLD  that 
better  represents  the  inherent  social  dilemma  and  other  central  aspects  of  the  joint  program 
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dilemma  as  seen  by  the  workshop  participants.  Appendix  B  contains  this  refined  CLD.4  As 
additional  workshops  are  conducted,  other  aspects  may  be  included  or  excluded  from  the 
CLD  based  on  the  findings  of  those  workshops. 

The  only  loops  retained  in  this  simpler  model  are  the  stakeholder  custom 
requirements  acceptance  (B3),  pressure-induced  rework  (R3),  and  pressure-induced 
attrition  (R4),  as  described  in  Appendices  A  and  B.  The  first  two  of  these  were  the  top  two 
rated  feedback  loops  at  the  workshop.  The  third,  which  is  closely  related  to  the  second, 
occurs  in  most  joint  programs  and  causes  significant  turmoil  and  lost  productivity.  Also 
included  is  one  of  the  two  highest  rated  extensions  proposed  to  the  original  model:  The 
inclusion  of  Joint  Program  Office  (JPO)  efforts  to  keep  the  joint  program  sold  to  stakeholders 
was  deemed  a  key  contributing  factor  to  endemic  problems  and  inefficiencies.  The  top- 
rated  extension  that  was  suggested  at  the  workshop,  the  distinction  between  acquiring 
capabilities  as  opposed  to  acquiring  systems,  will  be  addressed  explicitly  in  future  versions 
of  the  model. 

The  system  dynamics  method  provides  a  way  of  implementing  a  CLD,  so  as  to 
further  explore  the  implications  of  the  causal  structure  as  it  is  elaborated  in  more  detail. 
These  implications  are  assessed  through  simulation  (execution)  of  the  model.  In  addition  to 
the  confidence  gained  in  the  CLD  during  the  workshops,  simulation  can  result  in  additional 
confidence  that  the  causal  structure  can  indeed  produce  the  behavior  implied  by  the 
qualitative  CLD.  Once  the  model  has  been  shown  to  exhibit  the  expected  behavior, 
workshop  interactions  can  help  ensure  that  it  does  so  for  the  correct  reasons.  This  level  of 
validation  then  allows  the  analyst  to  use  the  model  to  test  alternate  solutions  to  the  problem 
using  the  system  dynamics  simulation  capability. 

The  simulation  and  analysis  of  the  joint  program  model  is  still  ongoing,  and  it  is  the 
initial  results  of  that  effort  that  are  presented  here.  The  feedback  that  was  received  in  the 
initial  problem  elaboration  workshop  made  it  possible  to  simplify  and  focus  the  original 
simulation  model  that  had  been  developed.  The  three  primary  segments  of  the  current 
simulation  model  are  described  in  order:  the  Stakeholder  Program  Segment,  the  Joint 
Program  Office  (JPO)  Segment,  and  the  Developer  Segment.  Each  of  the  stakeholder 
programs,  the  JPO,  and  the  developer  have  reasons  to  be  at  least  comparatively  satisfied 
based  on  the  progression  of  events  thus  far,  early  on  in  the  joint  program  acquisition. 
However,  as  will  be  seen  in  the  subsequent  section,  Systemic  Effects,  their  relative 
satisfaction  will  be  spoiled  due  to  the  diminishing  returns  associated  with  joint  program 
expenditures. 

•  The  current  model  makes  the  following  assumptions  about  the  joint 
acquisition  program: 

•  The  timeline  of  the  simulation  is  1 20  months — 1 0  years — but  the  conclusion 
of  the  project  may  be  significantly  short  of  that,  and  vary  depending  on  the 
input  parameters.  Milestone  B  occurs  12  months  into  the  simulation,  and  that 
is  when  the  development  contract  is  awarded. 

•  The  joint  program  has  three  stakeholder  programs  that  negotiate  with  the 
JPO  for  their  own  custom  requirements  separate  from  a  set  of  baseline 
requirements.  The  stakeholder  programs  are  referred  to  abstractly  as  SI,  S2, 
and  S3. 


4  Note  that  CLDs  and  system  dynamics  models  share  a  similar  notation.  The  primary  difference  is  that 
CLDs  do  not  include  stocks  or  flows.  They  are  strictly  qualitative  and  so  are  not  executable. 
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•  Funding  for  the  joint  program  is  spent  strictly  on  development  activities.  JPO 
staff  can  rotate  out  and  be  hired  in,  but  the  staff  levels  stay  at  generally  the 
same  level  and  do  not  consume  funding  (e.g.,  they  are  on  overhead,  as  far  as 
the  model  is  concerned). 

•  Developer  staff  are  separated  into  new  staff  versus  experienced  staff,  each 
with  their  own  levels  of  productivity  (i.e.,  computer  software  configuration 
items  (CSCIs)5  developed/tested  per  month)  and  monthly  costs.  Experienced 
staff  may  have  their  time  partially  consumed  by  training  new  staff. 

These  assumptions  may  be  relaxed  in  future  revisions  to  the  model  to  allow  a  broader  range 
of  behaviors  to  be  tested. 

It  should  be  noted  that  although  the  model  described  as  follows  has  been  refined 
both  by  the  problem  elaboration  workshop  sessions  and  through  the  acquisition  experience 
of  the  modeling  team  itself,  this  model  has  not  yet  been  validated  with  historical  joint 
program  data  to  help  quantify  the  relationships  between  the  model  variables.  This  validation 
will  be  conducted,  but  at  this  point,  the  model  should  be  viewed  as  providing  only  tentative 
support  for  the  causal  hypothesis. 

Stakeholder  Program  Segment 

A  primary  concern  of  the  stakeholder  programs  is  getting  their  (custom)  requirements 
implemented  by  the  joint  program  so  that  they  have  the  most  usable  system  possible  when 
the  joint  program  completes  development.  There  is  a  fair  amount  of  negotiation  going  on 
during  these  times  between  the  joint  program  office  and  the  stakeholder  programs,  and  the 
initial  model  is  based  on  the  foundations  of  negotiation  and  cooperation  theory.  Other  work 
in  developing  system  dynamics  models  has  leveraged  some  of  this  theory  in  the  past.  This 
model  is  based  explicitly  on  models  developed  by  Darling  and  Richardson  (1990). 

As  illustrated  in  Figure  2,  stakeholder  program  decision-making  is  based  on  the 
following: 

•  Stakeholder  program  gain  (the  inner  loop  in  the  figure).  The  extent  to  which 
the  stakeholder  program’s  custom  requirements  are  implemented  in  the  joint 
system.  In  terms  outlined  by  Darling,  this  gain  limits  the  stakeholder 
program’s  problem  potential.  An  effect  function6  is  used  to  capture  the 
framing  effects  of  Darling’s  model,  which  is  used  to  determine  whether  the 
extent  of  the  stakeholder  program’s  gain  is  viewed  positively  or  negatively. 

•  Stakeholder  program’s  relative  gain  (the  outer  loop  in  the  figure).  The 
stakeholder  program’s  satisfaction  is  also  dependent  on  how  much  they 
perceive  others  are  gaining  relative  to  their  own  gain.  If  they  think  others  are 
getting  proportionally  more,  then  they  will  be  less  satisfied  even  if  they  are 
still  getting  their  own  needs  met  adequately.  This  is  a  refinement  of  Darling’s 
model,  which  was  based  on  a  weighted  sum  of  the  gain  for  self  and  the 
perceived  gain  of  other  stakeholder  programs. 

o  A  more  recent  perception  of  gains  weighs  more  in  stakeholder  program 
decision-making  than  older  perceptions.  This  relates  to  the  moving 
average  used  in  the  Darling  model,  which  models  how  past  outcomes 
influence  present  expectations. 


5  A  CSCI  is  a  collection  of  software  that  supports  a  specific  function  for  the  end  user. 

6  An  effect  function  is  a  device  used  in  system  dynamics  modeling  that  explicitly  describes  the 
mathematical  relationship  between  two  specific  model  variables  overtime. 
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o  The  possibility  that  a  stakeholder  program  may  have  only  a  limited 
understanding  of  other  stakeholder  programs’  gains  (Darling’s  “Fixed 
Pie  Bias”)  is  handled  with  a  weighted  formula.  To  the  extent  that 
understanding  is  incomplete  (i.e.,  knowledge  of  other’s  gain  is  less  than 
1),  a  stakeholder  program  assumes  that  their  loss  is  the  other 
stakeholder  program’s  gain. 


Initial  discussion  with  joint  program  decision-makers  suggests  that  concern  for 
fairness,  as  described  in  Darling’s  model,  is  not  a  primary  factor  in  stakeholder  program 
decision-making,  so  it  has  been  omitted  from  the  simplified  model  presented  here.  It  is, 
however,  still  a  factor  in  the  larger  model  being  developed. 

A  stakeholder  program’s  satisfaction  influences  both  the  extent  of  their  buy-in  to  the 
joint  program  and  their  cooperation  with  the  joint  program  goals.  Both  buy-in  and 
cooperation  with  the  joint  program  are  needed  to  keep  the  program  viable.  When  either  is 
lagging,  the  JPO  will  tend  to  implement  more  of  the  stakeholder  program’s  custom 
requirements  to  keep  the  stakeholder  program  engaged. 


_ satiiiicaoa _ 

Figure  2.  Stakeholder  Programs  Negotiate  for  Custom  Requirements  Beyond  Baseline 

This  effect  can  result  in  an  escalation  of  custom  requirements,  which  of  course  must 
then  be  integrated  with  the  original  requirements.  The  model  initial  settings  are  set  to  an 
equilibrium.  At  Month  18,  to  test  the  behavior  of  the  model,  the  demands  of  stakeholder  SI 
are  stepped  up  to  a  level  of  0.8  on  a  scale  of  0  to  1 .  This  perturbation  from  equilibrium 
shows  in  Figure  3  that  increases  in  one  stakeholder  program’s  demands  leads  to  increases 
in  other  stakeholder  programs’  demands.  Although  the  levels  do  not  rise  to  the  same 
degree,  the  escalation  of  custom  requirements  that  result  are  necessary  from  the  joint 
program  perspective  in  order  to  maintain  stakeholder  programs’  buy-in  and  prevent 
stakeholders  from  defecting. 
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Fraction  of  Custom  Requirements  that  are  Accepted  by  die  JPO 
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&»ct!oncujiomfeqiicceped[S2] :  Qssr ea  - 2 - 2 - 2 - 2 - 

friction cu«onireqs»ccepe<l{S3j :  Oarea  -3 - 3 - 3  3  3 — 


Figure  3.  Increase  in  Custom  Requirements  Acceptance  for  SI  With  Subsequent  Rise  for 

S2  and  S37 

In  the  Darling  model,  this  behavior  reflects  the  “competitive  drift”  possible,  where  one 
negotiator  is  pitted  directly  against  another  and  the  interaction  between  negotiators 
becomes  increasingly  acrimonious.  In  the  joint  program  case,  the  JPO  may  feel  compelled 
to  give  in  to  stakeholder  program  demands  across  the  board,  directly  supporting  the  creation 
and  reinforcement  of  the  underlying  social  dilemma.  With  greater  support  being  given  to 
their  individual  needs,  the  stakeholder  programs  remain  relatively  satisfied. 

Joint  Program  Office  (JPO)  Segment 

The  benefit  of  keeping  stakeholder  programs  “bought  in”  to  the  joint  program  is 
evident  in  Figure  4.  More  engaged  program  stakeholders  promote  DoD  buy-in.  Once  the 
development  starts,  especially  with  the  additional  custom  requirements  accepted,  plus-ups 
on  funding  and  extensions  to  the  schedule  are  usually  necessary  to  implement  the 
additional  functionality. 


7  This  and  subsequent  graphs  were  generated  using  the  Vensim  modeling  tool.  These  are  ail 
behavior-over-time  graphs,  and  as  such,  the  x-axis  for  these  graphs  is  specified  in  months  (120 
months — 10  years — is  the  duration  of  this  simulation).  Each  simulation  run  is  specified  as  individual 
graphs  distinguished  with  a  number  label  (1  through  3  in  Figure  3),  as  specified  in  the  legend  below 
the  graph. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  109- 


Figure  4.  JPS  Benefits  From  Increased  Stakeholder  Program  Buy-In  by  Keeping  the 

Program  Alive 


Total  Funding 


0  12  24  36  48  60  72  84  96  108  120 

Time  (\bnth) 


Total  Funding  :  Current  -1 - 1 - 1 - 1 - 1 1 1 - 4 

Total  Funding:  Baseline  —2 - 2 - 2 - 2 - 2 - 2 - 2 - 


Figure  5.  Additional  Funding  Increments  to  Implement  Expanded  Scope 
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Announced  Completion  Date 


Announced  Completion  Date  :  Current  “A - t - t - t - 1 - E 

.Announced  Completion  Date  :  Baseline  — 2 - 2 - 2 - 2 - 2 - 


Figure  6.  Additional  Schedule  Extensions  to  Implement  Expanded  Scope 


Developer  Segment 

The  additional  development  work  generated  due  to  the  additional  custom 
requirements  from  the  stakeholder  platforms  is  shown  in  the  middle  of  Figure  7.  This 
additional  development  work,  along  with  the  development  work  from  the  originally  planned 
baseline,  is  added  to  the  development  work  remaining.  Both  development  and  testing  work 
is  accomplished  based  on  the  productivity  of  the  development  staff,  shown  on  the  left  side  of 
the  figure. 


Figure  7.  Development  Staff  Managed  to  Complete  Development  Work 


Development  staff  is  split  between  new  hires  and  experienced  staff,  with  some 
training  period  (possibly  on-the-job)  needed  to  transition  from  new  to  experienced.  The 
development  productivity  levels  of  new  and  experienced  staff  differs,  with  experienced  staff 
spending  some  of  their  time  training  the  newer  staff.  All  charges  made  by  the  staff  for  their 
time  working  on  the  project  are  reflected  in  the  cumulative  contractor  (i.e. ,  developer) 
revenue.  As  shown  in  Figure  8,  the  contractor’s  revenue  rises  well  above  the  baseline 
levels,  partially  due  to  implementing  the  additional  custom  requirements  demanded  by  the 
stakeholder  programs.  In  this  context,  assuming  that  the  contractual  negotiations  are 
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providing  additional  revenue  for  the  additional  employees,  the  contractor  is  willing  (if  not 
even  happy)  to  employ  more  staff  for  a  longer  period. 


Cumulative  Developer  Revenue 


Cumulative  Developer  Revenue  :  Current  - 1 - 1 - 1 1 - !■ 

C  mini  ative  Developer  Re  venue  :  Baseline  - 2  2 - 2 - 2 - 


Figure  8.  Developer  Revenue  Rises  Well  Above  the  Baseline  Level8 
Systemic  Effects 

Although  the  stakeholder  programs,  the  JPO,  and  the  developer  accomplish 
important  objectives  in  their  own  domains,  these  objectives  act  as  a  trap  for  joint  program 
decision-makers  that  can  potentially  lead  to  the  demise  of  the  joint  program.  Figure  9  shows 
the  diminishing  returns  related  to  the  joint  program  investment  to  develop  the  extended  joint 
system.  As  the  number  of  custom  requirements  accepted  for  each  stakeholder  program 
increases  along  the  x-axis,  the  average  cost  per  CSCI  increases  by  a  factor  of  5  to  10  over 
the  average  cost  per  CSCI  in  the  baseline  development.  Another  dimension,  along  the  z- 
axis,  shows  that  as  the  realism  of  schedule  setting  decreases  from  1  to  0.1 ,  the  CSCI  cost 
ratio  declines  even  further.  As  a  result  of  the  simulation  and  analysis  of  this  scenario, 
representations  of  complex  decision  surfaces  such  as  shown  in  Figure  9  allow  decision¬ 
makers  to  understand  the  interactions  between  multiple  factors  within  a  system,  and  to 
understand  the  range  of  possible  outcomes  based  on  various  actions. 


8  Development  is  complete  about  Month  30  in  the  Baseline  simulation  run  and  about  Month  47  in  the 
Current  run. 
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Figure  9.  Systemic  Result:  Diminishing  Returns  in  Development  Effort  Lead  to  Cost 

Increases  for  Program 

The  overview  model,  shown  in  Figure  10,  integrates  the  stakeholder  program 
segment,  the  JPO  segment,  and  the  developer  segment  described  previously.  The  model 
also  illustrates  the  primary  influences  causing  the  diminishing  returns: 


•  Complexity-Induced  Rework  (in  blue  in  the  lower  middle  of  the  figure) — The 
system  complexity  that  results  from  program  stakeholder  custom 
requirements  decreases  average  development  productivity  and  increases  the 
rates  of  defect  injection  during  development.  The  increased  system 
complexity  increases  the  complexity  of  developing  individual  CSCI  for  a 
variety  of  reasons,  making  development  take  longer  and  be  more  error  prone. 

•  JPO  Staffing  Effects  on  Program  Execution  (in  green  in  the  lower  middle  of 
the  figure) — The  resource  demands  on  the  JPO  staff,  as  described  previously 
in  the  JPO  segment,  causes  two  primary  problems  for  the  developers.  First, 
the  JPO  staff  is  not  as  responsive  to  developer  demands  for  guidance,  and 
for  review  and  feedback  on  development  artifacts.  This  reduces  the  average 
developer  productivity.  The  second  effect  is  that  the  JPO  staff  shortcuts  the 
quality  of  their  guidance  and  review  process.  This  leads  to  lower  quality  in  the 
development,  and  greater  amounts  of  rework. 

•  Pressure-Induced  Rework  (the  red  reinforcing  feedback  loop) — The 
expansion  of  the  joint  system  scope  leads  to  the  need  for  extensions  to  the 
schedule  well  beyond  those  planned  for  the  original  baseline  system. 

Although  the  need  for  schedule  extensions  is  widely  recognized,  they  may 
come  infrequently  at  unpredictable  times,  and  only  if  decision-makers  remain 
adequately  bought  in.  The  result  is  intense  schedule  pressure,  which  may  be 
evident  even  early  in  the  program  if  the  initial  schedule  was  unrealistic.  Such 
schedule  pressure  can  lead  to  bypassing  some  quality  processes,  and  to  the 
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generation  of  higher  levels  of  rework.  This  acts  in  a  reinforcing  manner  as 
schedule  pressure  escalates  even  further. 

•  Pressure-Induced  Attrition  (the  purple  reinforcing  feedback  loop) — 
Development  staff  may  suffer  the  most  from  schedule  pressure.  When 
development  staff  are  in  high  demand,  attrition  may  grow.  Despite  new 
development  staff  being  hired,  the  average  and  thus  the  overall  productivity 
may  fall,  making  it  even  harder  to  meet  schedule  demands.  This  reinforcing 
dynamic  exacerbates  the  problem  further. 

This  section  described  the  hypotheses  about  why  joint  programs  can  get  trapped  into 
a  development  of  diminishing  returns.  The  four  causes  for  these  diminishing  returns, 
described  previously,  provide  a  view  of  what  can  go  wrong.  Mitigation  of  this  problem  may 
involve  developing  a  means  to  avoid  falling  into  the  trap  in  the  first  place  or  for  reducing  the 
negative  consequences  associated  with  falling  in  the  trap.  The  next  section  describes  some 
of  the  considerations  regarding  problem  mitigation. 

Mitigations  for  the  Joint  Program  Dilemma 

The  rationale  for  identifying  a  possible  inherent  social  dilemma  at  work  within  the 
structure  of  a  joint  program  is  to  understand  the  mechanism  by  which  these  types  of 
acquisition  programs  can  encounter  difficulties.  Once  the  mechanism  has  been  confirmed, 
there  is  a  large  set  of  mitigations  and  solution  approaches  that  have  been  developed  in 
different  academic  disciplines  such  as  game  theory,  behavioral  economics,  social  science, 
and  social  psychology,  with  each  addressing  differences  in  the  specifics  of  the  instance  of 
the  dilemma.  Elinor  Ostrom  received  the  2009  Nobel  Prize  in  Economics  for  her  study  of 
innovative  solutions  that  have  evolved  in  different  cultures  to  address  differing  instances  of 
the  “Tragedy  of  the  Commons.”  However,  these  academic  solutions  are  not  well  known  to 
the  software-intensive  acquisition  or  software  development  communities  and  thus  have  not 
yet  been  studied  in  the  context  of  acquisition  programs,  so  their  applicability  is  still  unknown. 
The  goal,  however,  remains  the  same — to  deploy  higher  quality  systems  to  the  field  in  a 
more  timely  and  cost-effective  manner. 

The  research  literature  organizes  the  solutions  to  social  dilemmas  such  as  the 
“Tragedy  of  the  Commons”  into  three  classes: 

•  Motivational.  Motivational  solutions  assume  that  participants  are  not  exclusively 
self-interested  and  thus  care  about  the  consequences  of  their  actions  on  other 
participants.  Because  of  this,  such  concerns  as  values  and  group  identity,  as  well 
as  communication,  can  be  effective. 

•  Strategic.  Strategic  solutions  assume  that  participants  are  exclusively  self- 
interested  and  so  require  that  the  participants  influence  how  the  other 
participants  behave,  thus  producing  a  better  outcome  for  themselves.  Robert 
Axelrod  (1984)  provided  three  ingredients  for  such  approaches:  (1)  long-term 
relationships  among  the  participants  (so  that  all  expect  shared  dilemmas  in  their 
future),  (2)  that  the  participants  can  identify  one  another,  and  (3)  that  participants 
are  aware  of  the  past  behavior  of  each  other. 

•  Structural.  Structural  solutions  require  changing  the  rules  of  the  situation  so  that 
the  nature  of  the  dilemma  also  changes.  The  most  significant  difficulties  with 
applying  structural  solutions  is  that  (1)  they  require  a  level  of  authority  to 
implement,  (2)  they  may  bring  about  resistance  from  those  who  are  affected,  and 
(3)  they  require  methods  (with  accompanying  costs)  to  ensure  compliance  with 
the  new  rules. 
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The  first  two  classes  (i.e.,  motivational  and  strategic)  do  not  require  changing  the 
fundamental  structure  of  the  situation,  and  as  a  result,  they  tend  to  be  simpler  to 
implement — although  their  effectiveness  is  less  certain  when  compared  to  a  structural 
solution. 

To  discuss  one  common  approach  to  resolving  a  social  trap,9  the  use  of  an  authority 
to  manage  the  commons  is  widely  used  in  practice.  However,  this  approach  may  have  side 
effects,  depending  on  how  the  leader  was  selected  and  from  which  organization,  since  the 
perceived  objectivity  and  neutrality  of  the  leader  is  essential  to  their  acceptance  by  the 
participants. 

Another  widely  used  approach  is  privatization,  which,  like  the  use  of  authority,  also 
has  side  effects.  By  removing  the  social  aspect  of  the  social  dilemma,  it  eliminates  the 
interdependence  between  people  by  converting  shared  ownership  to  private  ownership. 
However,  this  would  result  in  each  of  the  stakeholder  programs  building  their  own  custom 
system,  which  is  antithetical  to  the  originally  intended  outcome. 

Another  approach  that  could  produce  a  better  outcome  might  be  altruistic 
punishment  (Fehr  &  Gachter,  2002).  In  altruistic  punishment,  cooperating  participants  may 
penalize  uncooperative  participants  through  some  mechanism  (such  as  withholding  a  small 
funding  increment)  at  a  small  cost  to  themselves.  Participants  seem  willing  to  do  this, 
despite  the  cost — and  even  if  it  will  yield  no  direct  material  gain  to  them.  Fehr  and  Gachter’s 
research  indicated  that  cooperation  increases  if  altruistic  punishment  is  possible  and  may 
break  down  if  it  is  ruled  out.  In  addition,  imposing  a  cost  on  the  administering  party 
disincentivizes  overuse,  making  it  self-correcting. 

Such  a  solution  could  help  to  avoid  the  requests  for  additional  capabilities  and 
prevent  the  downward  spiral  due  to  a  lengthening  schedule  and  increasing  cost,  complexity, 
and  risk,  thus  incentivizing  stakeholder  programs  to  stay  with  the  joint  program,  rather  than 
defect.  However,  this  particular  solution  to  the  social  trap  may  or  may  not  be  feasible  for  use 
on  a  joint  program. 

Another  way  of  addressing  a  social  trap  would  be  a  strategic  approach:  making  a 
series  of  small  changes  to  the  incentive  and  reward  structure  of  the  program,  such  as 
improving  communications,  making  negative  behaviors  more  visible  to  all  participants,  and 
similar  modifications.  Although  no  single  such  change  would  be  likely  to  significantly  mitigate 
the  problem,  it  may  be  that  the  aggregate  effect  of  many  small  changes  to  the  program 
structure,  when  taken  together,  could  have  a  substantial  positive  impact. 

Other  solutions  to  addressing  social  dilemmas  exist,  such  as  building  trust,  exclusion 
mechanisms,  assurance  contracts,  and  many  others.  The  choice  of  the  best  solution  will 
depend  on  the  specific  circumstances  surrounding  the  specific  joint  program  dilemma. 

The  defense  acquisition  system  itself  poses  some  significant  challenges  to 
successfully  mitigating  the  types  of  problems  that  are  inherent  to  joint  programs  and 
common  infrastructure  programs.  When  looking  at  the  structural,  strategic,  and  motivational 
classes  of  solutions  to  social  dilemmas,  it  is  apparent  that  motivational  solutions,  while 
attractive  due  to  their  generally  lower  level  of  effort  to  implement,  may  have  little  ability  to 
effect  change  if  the  participants  have  substantial  self-interest.  The  knowledge  that  “the 
complicated  acquisition  system  generates  staggering  bargaining  and  coordination  costs” 
such  as  “bureaucratic  politics  including  inter-service  rivalry,  Joint  service  logrolling”  (Lindsay, 
2006)  make  a  belief  in  the  services  having  low  levels  of  self-interest  seem  unlikely.  Strategic 
solutions  are  more  pragmatic  but  rely  largely  on  the  reputation  of  individuals  and  longer  term 

9  Social  traps  were  discussed  in  the  section  Misaligned  Incentives  in  Acquisition. 
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relationships  between  negotiating  parties,  both  of  which  are  in  short  supply  in  a  military 
where  “the  average  tenure  for  program  management  in  DoD  is  only  18  months”  (McConnell, 
Sickler,  &  Yang,  2004).  Structural  solutions  thus  may  appear  to  have  the  most  promise  of 
the  three  classes,  although  convincing  all  of  the  authorities  required  both  to  implement  and 
enforce  new  rules  on  all  parties  in  a  joint  program  context  may  prove  to  be  problematic. 

The  research  with  the  system  dynamics  model  of  joint  programs  that  is  being 
developed  involves  the  selection  of  some  of  the  most  promising  mitigation  and  solution 
approaches,  and  modeling  those  approaches  in  the  context  of  the  joint  program  model.  By 
assessing  the  ability  of  these  solution  approaches  to  mitigate  the  key  adverse  dynamics  that 
are  often  present  in  joint  programs,  it  will  be  possible  to  identify  a  set  of  the  most  promising 
approaches  that  could  be  applied  in  practice  to  try  to  avoid  these  issues  in  an  actual  joint 
acquisition  program. 
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Future  Work 

Some  of  the  possible  areas  for  future  work  will  involve  additional  refinement  and 
validation  of  the  simulation  model  through  review  and  feedback  by  joint  acquisition  domain 
experts,  as  well  as  calibration  with  historical  program  performance  data.  Once  sufficient 
confidence  in  the  model  is  gained  through  validation,  it  can  be  studied  further  to  understand 
how  the  different  key  model  variables  are  interrelated,  and  contribute  to  the  causes  of 
problematic  behaviors.  Complex  surfaces,  such  as  the  one  shown  in  Figure  9,  can  be 
created  to  give  a  sophisticated  understanding  to  decision-makers  as  to  how  multiple 
variables  interrelate  and  interact.  The  model  can  thus  be  used  as  a  management  decision 
aid  to  gain  an  understanding  of  what  might  occur  in  the  future  if  current  conditions  continue 
unchanged  and  to  explore  hypothetical  “what  if”  scenarios  based  on  potential  decisions  and 
events.  As  the  work  proceeds,  candidate  motivational,  strategic,  and  structural  mitigations  to 
the  problematic  dynamics  of  the  joint  program  social  dilemma  will  be  developed  and 
simulated  to  assess  their  effectiveness  and  viability,  and  to  help  develop  potential  new 
approaches  and  even  policy  recommendations  to  help  improve  the  execution  of  these  types 
of  programs. 

Although  no  model  can  accurately  predict  with  consistent  accuracy  the  future  states 
of  a  complex  dynamic  system  such  as  an  acquisition  program,  Donella  Meadows  (1974) 
pointed  out  that  “this  level  of  knowledge  is  less  satisfactory  than  a  perfect,  precise  prediction 
would  be,  but  it  is  still  a  significant  advance  over  the  level  of  understanding  permitted  by 
current  mental  models.” 

Conclusion 

This  paper  describes  the  results  of  a  preliminary  investigation  into  the  problems 
encountered  by  joint  acquisition  programs.  Through  interaction  with  joint  acquisition  experts, 
decision-makers,  and  stakeholders,  a  CLD  now  exists  that  represents  a  refined 
understanding  of  the  problem.  The  CLD  embodies  a  growing  comprehension  of  what 
happens  in  joint  acquisition  programs  and  why  the  stakeholder  programs,  the  JPO,  and  the 
developer  can  become  trapped  in  behaviors  that  make  rational  sense  to  the  participants  at 
the  time  but  can  lead  to  diminishing  returns  and  potentially  failure  for  the  program.  It 
describes  the  inherent  social  dilemma  that  exists  within  joint  programs — and  provides  the 
basis  for  a  better  understanding  of  the  problem  and  for  developing  ways  of  mitigating  it  to 
minimize  future  joint  program  challenges. 
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Appendix  A:  Feedback  Loops  Discussed  in  Workshop 

Table  1 .  Loops  of  the  Original  CLD  Discussed  at  the  Problem  Elaboration 

Workshop 


Loop 

Name 

Description 

R1 

Stakeholder 

Bandwagon 

Low  stakeholder  satisfaction  can  lead  to  a  desire  to  defect,  as  well  as  attempts  to 
influence  other  stakeholders  to  defect,  causing  a  vicious  cycle  that  can  collapse  the 
joint  program. 

B1 

Membership 

Management 

Lack  of  stakeholder  support  can  lead  to  low  service  support,  especially  if  the  program’s 
value  to  the  service  is  low.  This  may  require  a  greater  “marketing”  effort  by  the  JPO  to 
sustain  stakeholder  support. 

B2 

Program 

Support 

Lowered  service  support  can  undermine  DoD  support,  requiring  still  more  JPO 
“marketing”  effort  to  keep  the  stakeholders  engaged. 

R2 

Stakeholder 
Confidence  in 
JPO 

Stakeholder  support  can  grow  as  the  progress  of  the  grogram  adheres  to  the  schedule 
set  forth.  However,  if  the  program  falls  behind  schedule,  stakeholders  may  become 
dissatisfied,  start  to  lose  confidence,  and  ultimately  even  defect. 

B3 

Stakeholder 

Custom 

Requirements 

Acceptance 

Stakeholders  are  especially  concerned  with  meeting  their  own  custom  requirements.  To 
the  extent  those  requirements  are  not  addressed,  the  stakeholders  may  insist,  and  the 
JPO  may  eventually  need  to  accept  their  requirements. 

B3b 

Stakeholder 

As  more  of  a  stakeholder’s  custom  requirements  are  accepted,  fairness  to  others  may 
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Concern  for 
Fairness 

come  into  play.  The  stakeholder  may  become  more  cooperative,  lowering  his/her 
demands  for  more  custom  requirements. 

B4 

Honey  Rather 
than  Vinegar 

The  JPO  may  resist  accepting  custom  requirements  if  the  stakeholder  becomes  too 
demanding.  The  stakeholder  may  then  reassess,  becoming  more  cooperative  if  he 
thinks  more  of  his  custom  requirements  will  be  accepted. 

R3 

Pressure- 

Induced 

Rework 

Accepting  custom  requirements  leads  to  expanded  program  scope.  Without  schedule 
relief  or  additional  staff,  this  puts  additional  pressure  on  workers,  potentially  causing 
them  to  bypass  quality  processes,  thus  resulting  in  more  rework. 

B5 

De-scoping 

To  reduce  schedule  pressure  and  try  to  get  development  back  on  track,  the  JPO  may 
eliminate  requirements  or  defer  them  to  a  later  development  phase. 

R4 

Pressure- 

Induced 

Attrition 

If  sustained,  excessive  schedule  pressure  can  disgruntle  developers,  leading  to 
attrition,  and  making  it  even  harder  to  meet  schedule  demands. 

R5 

Stakeholder 
Missing  their 
Schedule 

Delaying  the  schedule  past  the  stakeholder’s  need  date  for  the  capability  increases 
dissatisfaction,  and  can  be  a  primary  cause  of  defection. 

R6 

Stakeholder 

Escalating 

Costs 

Expanding  project  scope  can  lead  to  greater  shared  costs  to  each  stakeholder.  This 
may  increase  discontent  and  lead  to  greater  demands  to  meet  custom  stakeholder 
requirements,  especially  early  on. 

Appendix  B:  Simplified  Causal  Loop  Diagram 
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Abstract 

This  research  examines  DoD  acquisition  from  the  context  of  a  network  of  interrelated 
programs  that  exchange  and  share  resources  for  the  purpose  of  establishing  joint 
capabilities.  The  research  focuses  on  the  joint  space  of  major  defense  acquisition  programs 
(MDAPs):  the  space  where  transactions  form  interdependencies  among  MDAP  programs. 

The  research  is  especially  salient  because,  to  date,  little  is  known  about  the  risks  associated 
with  interdependent  activities.  This  paper  provides  a  short  description  of  some  of  the  network 
characteristics  of  the  funding  and  data  interdependencies  of  major  defense  acquisition 
programs.  Where  the  discussion  focused  on  descriptions,  recent  advances  allow  the  ability 
to  test  the  structural  descriptions  on  program  performance.  In  exponential  random  graph 
models  (ERGM),  the  ties  serve  as  predictors  of  performance.  ERGMs  are  capable  of  testing 
a  host  of  structural  arrangements  for  their  influence  on  outcomes.  Employing  Markov  Chain 
Monte  Carlo  Maximum  Likelihood  Estimation  techniques,  probabilities  can  be  ascertained. 
Over  the  coming  months  the  structural  nature  of  the  interdependencies  will  be  analyzed  and 
evaluated  for  their  influence  on  acquisition  performance. 

Introduction 

In  a  world  of  insurgent  and  asymmetrical  warfare,  no  defense  organization  is  an 
island.  While  the  Services  have  engaged  in  a  host  of  coordinated  efforts  in  the  past,  the 
need  for  situational  awareness  and  rapid  response  rates  demands  the  synergistic  benefits 
that  only  wide-scale  cross-integration  and  interoperability  affords.  Never  in  the  history  of  the 
DoD  has  the  rapid  fielding  of  flexible  and  adaptive  technology  for  countering  unconventional 
and  time-sensitive  threats  been  more  important. 

This  research  examines  DoD  acquisition  from  the  context  of  a  network  of  interrelated 
programs  that  exchange  and  share  resources  for  the  purpose  of  establishing  joint 
capabilities.  The  research  focuses  on  the  joint  space  of  major  defense  acquisition  programs 
(MDAPs):  the  space  where  transactions  form  interdependencies  among  MDAP  programs. 
The  research  is  especially  salient  because,  to  date,  little  is  known  about  the  risks  associated 
with  interdependent  activities. 

Unfortunately,  by  and  large,  the  literature  on  interdependent  activities  is  steeped  in 
contradictory  findings.  For  example,  some  argue  that  tight-knit  arrangements  are  more  likely 
to  have  the  social  traction  needed  to  overcome  environmental  difficulties  (Sosa,  2011), 
whereas  others  argue  that  loose  coupling,  or  weak  ties,  may  be  a  better  solution 
(Granovetter,  1973).  Some  claim  that  more  information  is  the  key  to  benefit  attainment 
(Comfort,  1994),  whereas  others  claim  that  more  information  leads  to  a  false  sense  of 
security  (Hall,  Ariss,  &  Todorov,  2007).  Yet,  despite  the  absence  of  consistent  sage  advice, 
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resource  limitations  and  a  demand  for  comprehensive  solutions  continue  to  push 
organizations  toward  complex  structures  for  the  delivery  of  products  and  services. 

For  this  research,  jointness,  interdependency,  exchange,  and  partnerships  all  refer  to 
a  similar  concept:  the  notion  that  autonomous  organizations  build  relationships  to  obtain 
resources  to  provide  capabilities  that,  when  looked  at  in  totality,  form  network  structures. 
While  it  is  true  that  at  the  individual  pair-wise  level,  these  exchanges  exist  as  explicit 
transactions  for  the  transfer  of  data,  labor,  capital,  or  materials,  it  is  also  true  that  the  totality 
of  the  various  dimensions,  coupled  with  the  turbulence  of  perturbations,  influences  the  cost, 
schedule,  and  performance  of  the  acquisition  effort. 

Organizations  in  the  past  sought  to  limit  interdependencies  to  maintain  control  over 
the  environment.  More  recently,  however,  organizations  have  sought  to  leverage  the 
benefits  that  interdependencies,  or  partnerships,  can  provide.  Thus,  discussions  of  the 
nature  of  structure  and  how  to  best  organize  in  the  face  of  increasing  needs  for  holistic 
comprehensive  solutions  has  taken  center  stage.  The  key  question  seems  to  be  whether 
organizations  can  benefit  from  interdependence  while  minimizing  the  negative  influences  of 
environmental  turbulence.  The  question,  thus,  becomes,  what  structural  arrangements  and 
behavioral  practices  are  conducive  to  achieving  the  benefits  of  coordinated  actions?  The 
following  research  explores  the  nature  of  the  funding  and  data  interdependencies  that 
characterize  major  defense  acquisition  programs. 

Interdependent  Networks 

A  novice’s  glance  into  the  field  of  interdependent  organizational-based  networks  is 
likely  to  reveal  a  terminological  jungle  of  abstract  and  obscure  vocabulary.  This  section  of 
the  report  seeks  to  convey  many  of  the  more  common  network  terms  and  place  them  in  the 
context  of  DoD  acquisition.  Table  1  in  the  appendix  provides  a  glossary  of  several  of  the 
key  terms.  At  the  onset,  it  is  important  to  recognize  that  the  term  social  is  used  in  a  specific 
empirical  context  for  understanding  programmatic  interactions:  “Social  systems  of 
interaction”  form  the  basis  from  which  material  equipment  and  organizational  capacities  get 
things  done  (Turner,  1988). 

Wasserman  and  Faust  (1994)  defined  the  social  network  perspective  as  a  focus  on 
the  relationships  that  exist  among  entities  and  the  patterns  and  implications  of  these 
relationships.  Overall,  the  vantage  point  is  that 

•  actors  and  their  actions  are  viewed  as  interdependent  rather  than 
independent,  autonomous  units; 

•  relational  ties  between  actors  are  channels  for  the  transfer  of  resources;  and 

•  network  models  view  the  structural  environment  as  providing  opportunities 
for,  or  constraints  on,  individual  and  collective  action  (Wasserman  &  Faust, 
1994,  pp.  3-4). 

Organizations  have  long  been  viewed  as  resource  exchanging  agents.  When 
considered  in  this  light,  each  organization  takes  input  and  converts  it  into  outputs  that  are 
then  provided  as  inputs  to  another  organization.  Nonetheless,  in  the  past,  organizations 
often  sought  to  maintain  control  over  practices  and  procedures  by  restricting  access  to 
outside  influences.  Hierarchical  organizational  models  were  pursued  because  they  provided 
stability.  But  the  hierarchical  approach  was  found  to  be  ill-suited  to  situations  in  which 
needs  and  demands  evolved.  Hierarchical  approaches,  due  to  their  inability  to  adapt,  risked 
the  obsolescence  that  occurred  from  the  inability  to  adapt  to  changing  needs. 
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Over  the  years,  researchers  have  consistently  found  that  demand  uncertainty  is  a 
key  contributor  to  the  choice  to  forego  hierarchical-based  approaches  in  favor  of 
organizational  networks.  Demand  uncertainty  arises  when  organizations  lack  the  ability  to 
predict  near-future  needs.  When  organizations  are  confronted  with  high  levels  of  demand 
uncertainty,  they  require  the  flexibility  to  make  rapid  shifts  in  their  service  delivery  and 
production  cycles — shifts  that  a  hierarchical  approach  cannot  accommodate.  Because 
networks  offer  an  expanded  set  of  options,  they  allow  the  ability  to  respond  to  a  wider  range 
of  contingencies.  For  example,  under  asymmetric  warfare  conditions,  the  types  of  solutions 
that  may  be  required  are  difficult  to  predict  a  priori.  Given  the  uncertainty  of  the  demands  of 
the  battle  space,  warriors  require  a  wide  arsenal  of  alternative  and  complementary 
approaches — approaches  that  must  be  accessible  at  a  moment’s  notice.  When  demand 
uncertainty  is  low,  organizations  often  choose  more  simplistic  hierarchical  approaches. 
Under  high  demand  uncertainty,  organizations  require  the  ability  to  leverage  a  variety  of 
capabilities  irrespective  of  the  boundaries  of  a  given  organization’s  purview  (Jones, 

Hesterly,  &  Borgatti,  1997). 

In  the  work  setting,  network  actors  (or  nodes)  often  represent  people,  teams,  or 
organizations.  A  tie  represents  some  form  of  interaction  or  relationship.  In  short,  network 
structures  provide  the  “plumbing”  for  the  flow  of  resources  through  the  network. 
Interdependent  networks  are  complicated  by  the  fact  that  they  are  multidimensional,  and  as 

such,  understanding  their  behavior  requires 
consideration  of  multiple  levels  of  analysis. 
Typically,  networks  can  be  characterized  in  light 
of  four  basic  levels:  the  individual,  the 
subnetwork(s),  the  entire  network,  or  the 
multiplex  network.  A  multiplex  perspective 
considers  the  node  from  a  multi-network 
consideration.  For  example,  in  this  report,  major 
defense  acquisition  program  (MDAPs)  are 
examined  in  light  of  the  performance  of  the 
individual  program  as  well  as  its  resulting 
performance  in  two  different  networks:  (1)  a  data-sharing  network  and  (2)  a  shared  budget 
network.  Cross-level  effects  occur  when  behaviors  at  one  network  level  influence  behaviors 
at  another  network.  Cross-level  analysis  involves  looking  at  behavior  across  the  various 
networks.  The  failure  to  consider  cross-level  effects  may  result  in  misinterpreting  the  full  set 
of  consequences  that  occur  from  network  behaviors. 

At  the  individual  (or  node)  level,  an  ego  is  the  central  node  of  interest,  and  those 
connected  to  the  ego  are  known  as  alters  (see  Figure  2  in  the  appendix).  A  network 
rendering  from  the  context  of  an  ego  is  referred  to  as  an  ego-network.  A  dyad  consists  of  an 
ego  and  its  adjacent  alter.  As  discussed  further  in  the  next  section,  examining  data  in  light 
of  the  dyads  (or  pairs)  provides  the  ability  to  test  the  influence  that  one  node  has  on  another. 
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Directed  Network 


Undirected  Network 


# 

fork 


A  directed  network  is  one  where  the  flow 
of  resources  moves  in  a  specific  direction,  either 
inbound  to  an  ego  or  outbound  from  an  ego  (see 
Figure  3  in  the  appendix).  For  example,  the 
data-sharing  network  identified  previously  is  a 
directed  network  because  the  data  flow  from  one 
program  to  another.  A  directed  network  can  be 
either  sequential  or  reciprocal  in  nature. 
Alternatively,  an  undirected  network  is  one  that  is 
“pooled.”  In  other  words,  the  nodes  share  a 
lere  is  no  directional  component  to  the  tie.  In  this 


common  connection  (i.e. ,  a  budget),  butt 
case,  the  tie  indicates  that  the  two  programs  share  a  common  budget. 


A  node  is  labeled  as  a  broker  when  it 
connects  two  distinct  subnetworks.  So  in  Figure  4 
in  the  appendix,  Program  Number  554 
Multifunctional  Information  Distribution  System 
Joint  Tactical  Radio  System  (MIDS  JTRS)  acts  as 
a  broker  between  three  subnetworks.  An  isolate 
is  a  node  with  no  ties.  Again,  in  Figure  4  in  the 
appendix,  Program  Number  419  (EA  6B  Prowler) 
is  an  isolate.  In  directed  networks,  a  node  can 
serve  as  a  transmitter,  a  receiver,  or  a  carrier.  A 
bridge  is  identified  when  a  tie  spans  two 
subnetworks.  Structural  equivalence  occurs 
when  two  nodes  are  structurally  similar  (see 
Figure  5  in  the  appendix). 

Relying  on  matrix  algebra,  a  number  of 
metrics  have  been  devised  throughout  the  years 
to  measure  networks.  Some  of  the  metrics  occur 
at  the  node  or  ego  level,  and  others  are  at  the 
subnetwork  or  whole-network  levels.  Nodes  are 
often  considered  in  light  of  their  position,  or  role, 
in  the  network.  Many  of  the  ego-level  metrics  are 
calculated  relative  to  others  in  the  network. 


The  degree  of  a  node  is  the  number  of  ties 
that  a  node  exhibits.  These  ties  can  be  measured 
as  inbound  or  outbound  (or  both)  in  a  directed 
network.  Another  measure  is  the  geodesic 
distance  that  one  node  may  be  from  another. 
Adjacency  identifies  direct  connections  while 
reachability  identifies  whether  any  two  nodes  are 
capable  of  connecting  by  way  of  other  nodes. 
Degree  centrality  identifies  the  number  of  ties  that 
a  node  possesses.  The  more  ties  relative  to 
others,  the  greater  the  centrality.  Closeness,  on 
the  other  hand,  indicates  how  close  a  given  node 
is  to  the  remaining  nodes.  When  all  of  the  nodes  are  close  to  all  of  the  other  nodes,  the 
interaction  level  among  the  nodes  is  typically  high. 


1  &  2  are  structurally  equivalent 
3  &  4  are  structurally  equivalent 
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Network  size  is  often  calculated  as  the  sum  of  the  number  of  nodes  or  number  of  ties 
(see  Figure  6  in  the  appendix).  Sometimes  networks  (or  subnetworks)  are  measured  by 
their  longest,  or  shortest,  path.  The  bridge  identified  previously  is  often  of  interest  because 
it  indicates  that  if  the  tie  between  the  two  nodes  can  be  cut,  the  network  can  be 
disconnected  or  reduced  to  its  subnetworks.  The  same  holds  true  for  the  broker.  If  a  broker 
is  eliminated,  the  network  will  be  reduced  to  a  number  of  subnetworks.  Node  connectivity 
identifies  the  minimum  number  of  nodes  that  have  to  be  removed  to  disconnect  the  network. 
Betweenness  is  the  extent  to  which  a  given  node  lies  between  other  nodes  and,  thus,  could 
act  to  facilitate  or  block  the  flow  of  resources. 

Density  refers  to  the  proportion  of  ties  relative  to  the  absolute  total.  Relational 
embeddedness  refers  to  the  quality  and  depth  of  a  single  dyadic  tie.  Structural 
embeddedness  refers  to  the  extent  to  which  a  node’s  alters  are  connected  to  each  other. 
Because  structural  embeddedness  reflects  the  degree  of  the  interactions,  it  is  often  used  as 
a  proxy  for  understanding  network  actions. 

In  the  study  of  networks,  scholars  often  take  either  a  structural  or  a  connectionist 
approach.  Structural  approaches  examine  the  structure  of  the  network  and  its  influence  on 
key  variables  of  interest.  Connectionists,  on  the  other  hand,  focus  on  the  flows  between  the 
nodes.  Those  who  study  social  capital  tend  to  focus  on  the  possibilities  of  actions  that 
social  ties  provide.  Others,  however,  tend  to  be  more  concerned  with  diffusion  and  the 
dynamics  of  network  change  over  time.  Still,  other  studies  focus  on  why  and  how  networks 
develop,  how  and  why  they  change  over  time,  and  finally,  what  influences  they  exert.  Social 
capital  is  mostly  studied  at  the  individual  level,  and  diffusion  is  observed  from  the 
perspective  of  the  entire  network. 

Studies  of  the  influence  of  dyadic  ties  on  performance  have  mixed  and  contradictory 
findings.  For  example,  Perry-Smith  and  Shalley  (2003)  found  that  weak  ties  led  to  creativity, 
but  others  claim  that  strong  ties  are  more  advantageous  (Sosa,  2011).  Others  claim  that  it  is 
not  the  number  of  ties  but  rather  the  depth  of  the  engagement  that  matters.  No  one  would 
be  surprised  by  the  idea  that  relative  to  fewer  ties,  more  ties  may  provide  organizations  with 
better  information  that  might  promote  enhanced  decision-making.  At  the  same  time, 
information  overload  and  difficulties  with  scrubbing  data  to  provide  information  at  the  proper 
specification  level  have  become  real  problems  for  many  managers. 

Similarly,  studies  of  embeddedness  are  equally  contradictory.  According  to  some, 
the  more  each  node  knows  about  the  others,  the  more  constraints  there  are  on  each  other’s 
behaviors.  This  is  often  seen  as  a  positive.  Parties  gather  information  on  whom  to  avoid  as 
well  as  potential  opportunities  and  synergies.  Structural  embeddedness  allows  the  use  of 
sanctions  since  knowledge  of  misfeasance  influences  reputational  value.  But  these 
constraints  can  backfire  and  actually  restrict  flexibility.  Too  much  embeddedness  can  also 
create  problems.  It  can  lead  to  feuding,  group  think,  and  welfare  support  of  weak  members. 
Social  aspects  such  as  restricting  access  to  exchanges,  imposing  collective  sanctions,  and 
making  use  of  social  memory  and  cultural  processes  all  influence  nodal  behavior. 

Apparently,  networks  and  ties  matter,  but  the  extent  of  the  influence  is  highly  debatable. 

Much  of  the  incongruity  in  the  findings  may  be  due  to  the  difficulties  associated  with 
measurement  and  data  collection.  Researchers  are  challenged  by  the  burden  of  the  data 
collection  requirements,  and  organizations  are  often  frustrated  by  the  extent  of  the  data 
request.  Because  multilevel  data  are  needed  for  each  specific  relationship,  the  data 
collection  task  can  be  onerous.  Moreover,  given  that  the  study  of  networks  is  a  fairly  new 
phenomenon,  typical  organizational  records  often  lack  insights  at  a  network  level.  When 
multilevel  data  are  obtained,  an  analysis  of  variance  statistical  technique  termed  hierarchical 
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linear  modeling  or  multilevel  modeling  is  often  employed  because  it  allows  the  examination 
of  multiple  units  of  analysis  simultaneously. 

Despite  these  contradictory  findings  and  data  collection  difficulties,  the  examination 
of  networks  and  ties  that  manifest  as  interdependencies  is  likely  to  provide  substantial 
insights  into  a  number  of  issues.  First,  when  considering  cost  and  affordability,  examining  a 
program  in  isolation  of  the  entire  value  chain  is  likely  to  provide  erroneous  information. 
Second,  a  wealth  of  research  illustrates  the  importance  of  risk  management.  Considering 
the  risks  of  a  given  program  without  considering  its  interdependencies  may  underestimate 
the  true  risk  level.  Next,  in  the  decision  of  a  start-up  or  termination,  it  is  essential  to  know 
how  the  inclusion  or  removal  of  a  program  will  influence  its  n-order  neighbors.  Finally, 
network  conditions  may  exert  powerful  influences  over  program  sustainability.  The  following 
discussion  explores  the  funding  and  data  networks  employed  in  the  acquisition  arena. 

Interdependency  Descriptions 

Two  sets  of  interdependencies  are  examined  below.  One  set  reflects  funding 
interdependencies  and  the  other  captures  data  interdependencies.  In  the  organizational 
arena,  interdependencies  can  be  viewed  in  three  ways.  As  Thompson  (1967)  illustrates, 
network  arrangements  can  be  pooled,  sequential,  or  reciprocal.  Under  a  pooled 
arrangement,  network  actors  draw  down  from  a  common  pool  of  resources.  Under  this 
scenario,  the  actors  do  not  interrelate,  but  they  are  nonetheless  interdependent  because 
they  all  share  a  common  resource  that  can  be  depleted.  The  funding  interdependencies 
described  in  the  next  paragraph  reflect  a  pooled  relationship.  These  acquisition  programs 
share  a  common  program  element.  Thus  the  interconnections  reflect  their 
interdependencies  on  a  common  funding  source.  Sequential  relationships  are  often  termed 
supply  chains.  In  these  scenarios  resources  flow  in  a  sequential  manner  from  program  to 
program.  Reciprocal  relationships  are  often  seen  as  the  most  complex  and  have  the 
greatest  risk.  In  this  case,  resources  are  exchanged  and,  as  a  consequence,  there  is  a  two- 
way  link  among  the  programs. 

Figure  1  in  the  appendix  displays  the  funding  interdependencies  over  time.  As 
displayed  in  the  figure,  the  interdependencies  have  grown  increasing  complex  overtime. 

The  density  has  grown  from  a  low  of  6%  to  a  high  of  22%.  Figure  2  in  the  appendix  reflects 
the  polynomial  regression  equation  and  its  associated  bivariate  plot  showing  growth  over  the 
six-year  period.  Figure  3  in  the  appendix  illustrates  the  data  interdependencies.  As 
demonstrated  in  the  diagram,  these  interdependencies  reflect  326  ties  and  range  from  27% 
inbound  to  16%  outbound. 

Figures  4  and  5  in  the  appendix  illustrate  that  both  the  data  and  funding 
interdependencies  reflect  “preferential  attachment.”  Preferential  attachment  was  popularized 
by  Barabasi  and  has  gained  tremendous  attention  over  the  past  10  years.  Preferential 
attachment  (or  more  commonly  a  hub-and-spoke  model)  is  the  tendency  for  nodes  to 
establish  relationships  (or  links)  with  nodes  that  have  a  high  number  of  connections  with 
other  nodes.  As  a  result,  the  connections  demonstrate  a  power  law  distribution.  The  power 
law  distribution  is  important  because  it  illustrates  that  the  network  can  be  destroyed  by 
eliminating  the  “hubs.” 

Figures  6  and  7  in  the  appendix  show  the  funding  and  data  interdependencies  by 
Service  and  FCB.  As  shown,  the  Navy  appears  to  illustrate  the  greatest  number  of  funding 
and  data  interdependencies.  Interdependencies  by  FCB  appear  fairly  mixed. 
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Future  Activities 

This  paper  provides  a  short  description  of  some  of  the  network  characteristics  of  the 
funding  and  data  interdependencies  of  major  defense  acquisition  programs.  Where  the 
discussion  focused  on  descriptions,  recent  advances  allow  the  ability  to  test  the  structural 
descriptions  on  program  performance.  In  exponential  random  graph  models  (ERGM),  the 
ties  serve  as  predictors  of  performance.  ERGMs  are  capable  of  testing  a  host  of  structural 
arrangements  for  their  influence  on  outcomes.  Employing  Markov  Chain  Monte  Carlo 
Maximum  Likelihood  Estimation  techniques,  probabilities  can  be  ascertained.  Over  the 
coming  months,  the  structural  nature  of  the  interdependencies  will  be  analyzed  and 
evaluated  for  their  influence  on  acquisition  performance. 

References 

Ansoff,  H.  I.  (1979).  Strategic  management.  London,  UK:  The  MacMillan  Press. 

Cohen,  W.  M.,  &  Levinthal,  D.  A.  (1990).  Absorptive  capacity:  A  new  perspective  on  learning  and 
innovation.  Administrative  Science  Quarterly ,  35(1),  128-152. 

Comfort,  L.  (1994,  September).  Interorganizational  learning  following  the  Northridge  earthquake  of 
January  17,  1994.  Journal  of  Contingencies  and  Crisis  Management,  2(3), 174-188. 

Cyert,  R.  M.,  &  March,  J.  G.  (1992).  A  behavioral  theory  of  the  firm  (2nd  ed.).  Englewood  Cliffs,  NJ: 
Prentice  Hali.  (Original  work  published  1963) 

Granovetter,  M.  (1973).  The  strength  of  weak  ties.  American  Journal  of  Sociology,  78(6),  1360-1380. 

Grant,  R.  M.  (1996).  Toward  a  knowledge-based  theory  of  the  firm.  Strategic  Management  Journal, 

17,  109-122. 

Hall,  C.  C.,  Ariss,  L.,  &  Todorov,  A.  (2007).  The  illusion  of  knowledge:  When  more  information 
reduces  accuracy  and  increases  confidence.  Organizational  Behavior  and  Human  Decision 
Processes,  103,  277-290. 

Herbert,  S.  (1969).  The  sciences  of  the  artificial  (1st  ed.).  Cambridge,  MA:  MIT  Press. 

Jones,  C.,  Hesterly,  W.  S.,  &  Borgatti,  S.  P.  (1997,  October).  A  general  theory  of  network  governance: 
Exchange  conditions  and  social  mechanisms.  The  Academy  of  Management  Review,  22(4), 
911-945. 

Kauffmann,  S.  A.  (1993).  The  origins  of  order:  Self-organization  and  selection  in  evolution.  New  York, 
NY:  Oxford  University  Press. 

Lawrence,  P.  R.,  &  Lorsch,  J.  W.  (1967).  Differentiation  and  integration  in  complex  organizations. 
Administrative  Science  Quarterly,  12,  1-30. 

Lawrence,  P.  R.,  &  Lorsch,  J.  W.  (1967).  Organization  and  environment:  Managing  differentiation  and 
integration.  Cambridge,  MA:  Harvard  University,  Graduate  School  of  Business  Administration, 
Division  of  Research. 

Levinthal,  D.  (1997).  Adaptation  on  rugged  landscapes.  Management  Science,  43(7),  934-950. 

Levinthal,  D.,  &  Warglien,  M.  (1999).  Landscape  design:  Designing  for  local  action  in  complex  worlds. 
Organization  Science,  10(3),  342-357. 

March,  J.  G.,  &  Simon,  H.  A.  (1958).  Organizations.  New  York,  NY:  Wiley. 

McPherson,  J.  M.,  &  Ranger-Moore,  J.  (1991).  Evolution  on  a  dancing  landscape:  Organizations  and 
networks  in  dynamic  blau  space.  Social  Forces,  70, 19-42. 

Miles,  R.  E.,  &  Snow,  C.  C.  (1978).  Organizational  strategy,  structure,  and  process.  New  York,  NY: 
McGraw-Hill. 

Miles,  R.  E.,  &  Snow,  C.  C.  (1992,  Summer).  Causes  of  failure  in  network  organizations.  California 
Management  Review. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  127- 


Miles,  R.  E.,  Snow,  C.  C.,  &  Pfeffer,  J.  (1974).  Organization-environment:  Concepts  and  issues. 
Industrial  Relations,  13,  244-264. 

Milliken,  F.  (1987).  Three  types  of  perceived  uncertainty  about  the  environment:  State,  effect,  and 
response  uncertainty.  Academy  of  Management  Review,  12(1),  133-143. 

Perry-Smith,  J.  E.,  &  Shalley,  C.  E.  (2003).  The  social  side  of  creativity:  A  static  and  dynamic  social 
network  perspective.  Academy  of  Management  Review,  28(1),  89-106. 

Simon,  H.  A.  (1973).  Applying  information  technology  to  organizational  design.  Public  Administration 
Review,  33(3),  268-278. 

Simon,  H.  A.  (1976).  Administrative  behavior.  A  study  of  decision-making  processes  in  administrative 
organization  (3rd  ed.).  London,  UK:  The  Free  Press. 

Sosa,  M.  (2011).  Where  do  creative  interactions  come  from?  The  role  of  tie  content  and  social 
networks.  Organization  Science,  22(1), 1-21. 

Stacey,  R.  D.  (2002).  Strategic  management  and  organisational  dynamics:  The  challenge  of 
complexity  (3rd  ed.).  Harlow:  Prentice  Hall. 

Thompson,  J.  D.  (1967).  Organizations  in  action:  Social  science  bases  of  administrative  theory.  New 
York,  NY:  McGraw-Hill. 

Turner,  J.  (1988).  A  theory  of  social  interaction.  Palo  Alto,  CA:  Stanford  University  Press. 

Volberda,  H.  W.,  Foss,  N.  J.,  &  Lyles,  M.  A.  (2010).  Absorbing  the  concept  of  absorptive  capacity: 
How  to  realize  its  potential  in  the  organization  field.  Organization  Science,  21,  931-951. 

Wasserman,  S.,  &  Faust,  K.  (1994).  Social  network  analysis:  Methods  and  applications.  New  York, 
NY:  Cambridge  University  Press. 

Williamson,  O.  E.  (1981).  The  economics  of  organization:  The  transaction  cost  approach.  The 
American  Journal  of  Sociology,  87(3),  548-577. 

Acknowledgements 

This  material  is  based  upon  work  supported  by  the  Naval  Postgraduate  School  Acquisition 

Research  Program  under  Grant  No.  00244-10-1-0068. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  128- 


Appendix 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-  129- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-  130- 


Funding  Interdependencies  2009 


Links 


Funding  Interdependencies  2005 


-  Power 
(Funding 
Interdepe 
ndencies) 


Funding  Interdependencies  2006 

y  =  48.845x  1325 


-  Power 
(Funding 
Interdepe 
ndencies) 


5 

Links 


10 


Funding  Interdependencies  2007 


Links 


Funding  Interdependencies  2010 


Links 


Funding  Interdependencies  2011 


Links 


Figure  4.  Preferential  Attachment  of  Funding  Interdependencies 
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Data  Interdependencies 
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Figure  5.  Preferential  Attachment  of  Data  Interdependencies 
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Figure  7.  Data  Interdependencies  by  Service  and  FCB 
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Table  1.  Common  Network  Teams 


Node:  a  person,  team,  organization,  computer,  etc.,  in  a  network 

Tie:  a  connection  between  two  nodes 

Directed  Network:  a  network  where  the  tie  is  directional  in  nature 

Undirected  Network:  a  network  where  the  ties  are  not  directional 

Ego:  the  subject  of  the  discourse 

Alter:  the  node  that  the  ego  has  ties  with 

Ego  Network:  the  network  in  light  of  a  given  ego 

Dyad:  two  nodes  linked  into  a  pair.  Networks  can  be  decomposed  into  their  dyads,  or  pairs. 

Structuralist  Paradigm:  sees  the  network  structure  as  the  defining  characteristic  of  an  individual 
node’s  behavior.  By  extension,  two  nodes  that  share  structurally  similar  characteristics  will 
witness  similar  outcomes. 

Connectionist  Paradigm:  The  focus  is  on  the  resources  that  flow  through  the  ties;  the  ties  act  as 
conduits  for  the  flow  of  resources. 

Diffusion:  a  measure  of  the  spread  of  an  innovation  or  characteristic  throughout  the  network 

Social  Capital:  The  primary  focus  of  the  Connectionist  paradigm  is  concerned  with  the  resources 
that  are  gained  (or  lost)  via  the  ties,  and  it  views  success  as  a  function  of  these  ties. 

Structural  Capital:  The  primary  focus  of  the  Structuralist  paradigm  is  concerned  with  the  position 
of  nodes  in  a  network  and  how  this  influences  outcomes. 

Centrality:  the  extent  to  which  a  given  node(s)  dominates  the  number  of  ties.  When  only  a  few 
nodes  have  a  large  number  of  ties  compared  to  the  others,  the  network  is  viewed  as  highly 
centralized. 

Structural  Equivalence:  Actors  (or  nodes)  are  structurally  equivalent  to  the  extent  that  they  are 
similar  in  their  ties. 

Relational  Embeddedness:  relates  to  the  quality  and  depth  of  a  single  dyadic  tie 

Structural  Embeddedness:  relates  to  the  extent  to  which  a  given  node’s  alters  are  interconnected 

Geodesic  Distance:  represents  how  far  one  node  is  from  another.  It  is  often  represented  as  how 
near  or  far  a  node  is  from  another. 

Closure:  Is  a  measure  of  the  number  of  triads  (or  connections  among  three  nodes)  that  exist  in 
the  network 

Structural  Hole:  A  hole  in  the  network  that  a  node  could  bridge  and  thus  act  as  a  go-between.  In 
this  way,  structural  holes  can  often  control  the  two  nodes  that  they  connect. 

Broker:  Per  the  definition  of  structural  hole,  a  broker  spans  two  or  more  subnetworks. 

Multiplex  Ties:  when  a  given  node  connects  with  another  node  in  multiple  networks.  For 
example,  a  node  may  be  connected  to  another  node  in  both  a  funding  network  and  a  data-sharing 
network. 
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Homophily/Heterophily:  indicates  the  extent  to  which  one  node  is  similar  to  another  on  key 
characteristics 


Degree  Distribution:  the  variance  in  the  distribution  of  ties  in  a  network 


Network  Connectivity:  reflects  the  “size”  of  the  network  by  the  longest  path  from  one  node  to 
another 


Network  Density:  the  proportion  of  ties  in  a  network  relative  to  the  total  number  possible 


Pattern  of  Clustering:  refers  to  the  absence  or  presence  of  subnetworks 

Degree  Assortativity:  reflects  the  degree  to  which  nodes  with  a  similar  number  of  ties  connect 
with  each  other 


Cohesion:  the  degree  to  which  nodes  are  connected  directly  to  each  other.  Under  low  cohesion, 
a  number  of  cliques  (or  subnetworks)  will  be  observed. 

Bridge:  a  tie  that  is  critical  to  the  connectivity  of  the  network.  Elimination  of  the  bridge  is  likely  to 
result  in  a  large  number  of  factions. 


Path  Length:  the  length  from  one  node  to  another.  Typically  measured  in  terms  of  how  many 
nodes  are  in  between  the  two. 
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Abstract 

This  paper  describes  our  continuing  efforts  to  forge  new  ground  in  identifying  the  effects  of 
interdependency  on  acquisition  and,  if  needed,  uncovering  early  indicators  of 
interdependency  risk  so  that  appropriate  governance  oversight  methods  can  then  be  isolated. 
Specifically,  we  seek  to  study  the  topologies  of  Major  Defense  Acquisition  Programs 
(MDAPs)  networks  and  associated  cascading  consequences  of  interdependencies  in  such 
highly  dependent  networks.  Since  the  start  of  this  new  project  phase  a  couple  of  months  ago, 
we  have  begun  harnessing  the  extensive  data  that  has  been  collected  over  the  years  in  the 
form  of  Defense  Acquisition  Execution  Summary  (DAES)  documents  for  the  MDAPs.  We 
present  a  road  map  of  our  research  plan  and  our  preliminary  results  in  our  ongoing  efforts  on 
leveraging  network  structure  and  automatic  data  extraction  to  study  cascading  risks.  We  will 
also  identify  the  challenges  to  data  acquisition. 

Introduction 

This  research  seeks  to  study  the  structures  of  the  Major  Defense  Acquisition 
Programs  (MDAPs)  networks  and  the  associated  cascading  consequences  of 
interdependencies  in  such  highly  dependent  networks.  It  involves  identifying  the  effects  of 
interdependency  on  the  acquisition  process  and,  if  needed,  uncovering  early  indicators  of 
interdependency  risk  so  appropriate  governance  oversight  methods  can  then  be  isolated. 
Hence,  this  research  seeks  to  address  the  problem  that  there  is  little  insight  on  the  effects  of 
interdependencies  and  a  lack  of  tested  metrics  to  provide  early  indication  of  the  acquisition 
risks  of  interdependent  programs.  It  breaks  ground  in  the  area  of  (i)  studying  non-linear 
cascading  effects  in  the  context  of  a  network  of  MDAPs  consisting  of  some  not-so- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  137- 


successful  programs  (that  which  experiences  cost  growth)  as  compared  to  (ii)  the  study  of 
the  decision  mechanisms  of  successful  programs.  Lessons  learned  from  this  comparative 
analysis  would  help  model  the  behavior  of  other  MDAP  programs.  The  project  will  use  the 
extensive  data  that  we  have  collected  over  the  years  in  the  form  of  Defense  Acquisition 
Execution  Summary  (DAES)  documents  for  the  MDAPs. 

This  work  builds  on  our  previous  results  (Raja,  Hasan,  &  Brown,  2012)  obtained  from 
a  manual  analysis  of  data  belonging  to  a  small  network  of  MDAPs  representing  a  case 
study.  Our  goal  was  to  model  “what-if  analyses  that  would  help  decision-makers  to  gain 
insight  on  the  cascading  effects  of  perturbations  among  interdependent  networks  and  take 
appropriate  measures  to  handle  them.  We  used  the  case  study  to  first  determine  whether 
the  data  required  to  build  a  decision-theoretic  model  is  available  and  then  study  whether  this 
decision-theoretic  model  captures  the  cascading  interdependencies  that  are  of  interest  to 
us.  We  also  examined  the  data  investigation  process  to  identify  the  challenges  that  were 
encountered.  Our  results  showed  that  MDAP-related  data  characteristics  support  the 
multiple  perspective  study  of  perturbations  and  it  is  possible  to  recast  the  study  of  cascading 
effects  as  a  sequential  decision  problem.  We  identified  local  and  non-local  issues  that  when 
left  unmitigated  led  to  performance  breaches  in  the  MDAPs.  We  also  observed  that  it  is 
crucial  to  consider  the  uncertainty  in  action  outcomes  in  the  decision-making  process  and 
that  a  non-local  perspective  may  help  explain  a  performance  breach  in  situations  where  a 
solely  local  perspective  does  not.  These  observations  supported  our  conjecture  that  a 
decision-theoretic  model  is  a  good  methodology  to  study  interdependencies  in  the  MDAP 
network  and  to  capture  early  indicators  of  interdependency  risk.  Finally,  we  captured  the 
informational  value  in  the  existing  data  and  the  challenges  inherent  in  the  data  collection 
process  with  respect  to  their  role  in  isolating  risks  and  initiating  appropriate  government 
oversight  methods. 

The  sheer  volume  and  complexity  of  the  data  required  to  populate  our  decision- 
theoretic  models  effectively  has  led  us  to  identify  methods  for  automating  the  data 
extraction,  network  analysis,  and  construction  of  the  decision  model  that  is  the  focus  of  our 
current  work.  This  project,  initiated  a  couple  of  months  ago,  has  the  following  research 
goals:  (1)  Examine  and  compare  the  network  structure  characteristics  of  interdependent 
regions  belonging  to  successful  and  not-so-successful  MDAP  programs  to  augment  our 
current  work  in  “what-if”  analyses.  (2)  Automate  the  data  extraction  and  analysis  process  by 
leveraging  algorithms  for  decision  support  as  well  as  image  and  text  analysis.  (3)  Continue 
to  identify  the  challenges  in  acquiring  the  data  from  the  government  and  program  managers. 
In  this  paper,  we  will  discuss  our  proposed  ideas  for  this  year-long  project  and  the  initial 
work  we  have  done  to  achieve  the  above  mentioned  research  goals. 

Background 

It  has  been  shown  that  data  are  the  foundation  for  decision-making  in  the  acquisition 
environment.  The  Department  of  Defense  (DoD)  has  spent  a  significant  amount  of  effort 
working  across  the  organization  to  identify  useful  sources  of  data  and  to  conduct  analyses. 
The  importance  to  acquisition  research  of  studying  MDAP  interdependencies  was 
emphasized  during  the  2012  Annual  Acquisition  Research  Symposium  by  the  introduction  of 
a  new  panel  titled  Predicting  Performance  and  Interdependencies  in  Complex  Systems 
Development.  Prior  research  has  established  that  MDAPs  are  demonstrably  interdependent 
and  that  they  can  be  thought  of  as  networks  of  interdependent  programs  (Flowe,  Brown,  & 
Hardin,  2009;  Flowe,  Kasunic,  &  Brown,  2010;  Lewin,  1999).  Also,  the  acquisition  paradigm 
established  in  statute  (10  U.S.C.  2434;  Defense  Acquisition  Workforce  Act,  1990),  in  policy 
(DoD  5000.02;  Undersecretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
[USD(AT&L)j,  2008),  and  in  regulation  tends  to  favor  the  notion  of  MDAPs  as  being 
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independent,  which  would  cause  exogenous  factors  caused  by  interdependence  to  be 
overlooked  or  misinterpreted. 

Although  it  is  critically  important  to  understand  the  program  interfaces  and 
interdependencies,  there  are  few  tested  and  proven  tools  for  program  managers  and 
acquisition  executives  to  probe  the  joint  space  or  to  track  the  cascading  effects  that  the  joint 
space  might  trigger.  There  is  reason  to  believe  that  the  exogenous  issues  generated  from 
the  shared  domains  remain  unnoticed  to  the  extent  of  causing  the  program  to  potentially 
experience  severe  performance  degradation  (Brown,  2011).  The  complexity  of  the  joint 
environment  is  likely  to  have  a  direct  bearing  on  acquisition  activities.  The  precise  effect  on 
acquisition,  and  its  resulting  managerial  implications,  are,  as  of  yet,  unknown.  We  believe 
that  given  the  frequency  with  which  government  agencies  are  moving  toward  joint  initiatives, 
the  findings  of  this  research  project  based  on  DoD  programs  may  prove  instrumental  to  a 
wide-ranging  audience. 

Furthermore,  at  the  2012  Acquisition  Symposium,  Dr.  Frank  Kendall  III,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&Lj),  discussed  the 
DoD’s  strategic  priorities,  especially  around  acquisition.  These  priorities  included  achieving 
affordable  programs  that  execute  well  and  improving  efficiency  (via  Better  Buying  Power 
and  other  initiatives).  We  believe  the  work  described  in  this  paper  will  help  us  understand 
the  performance  of  the  programs  in  various  scenarios  and  contribute  directly  to  the  above 
priorities  by  achieving  affordable  programs  that  are  successful  as  well  as  improving  overall 
efficiency. 

Along  with  other  researchers  (Brown  &  Owen,  2012),  we  have  begun  to  harness  a 
network-centric  approach  to  study  DoD  acquisition  and  focus  on  an  MDAP  network  of 
interrelated  programs  that  exchange  and  share  resources  for  the  purpose  of  establishing 
joint  capabilities.  Some  work  (Zhao,  Gallup,  &  MacKinnon,  2012)  has  been  done  to  analyze 
the  unstructured  and  unformatted  acquisition  program  data  using  a  data-driven  automation 
system  called  lexical  link  analysis  (LLA).  LLA  is  used  to  determine  the  correlation  between 
system  interdependency  and  development  costs  in  an  effort  to  enable  acquisition 
researchers  and  decision-makers  to  recognize  important  connections  that  form  patterns 
derived  from  dynamic  data  collection.  In  other  work  (Han,  Fang,  &  DeLaurentis,  2012),  a 
Bayesian  Network  (BN)  method  is  used  to  assess  the  cascading  effects  of  requirement  and 
systems  interdependencies  on  risk  in  an  effort  to  effectively  analyze  alternatives  in  a 
capability-based  acquisition  strategy.  The  technique  is  evaluated  within  a  synthetic  network 
and  identifies  critical  systems  and  requirements. 

We  believe  our  work  will  help  us  understand  the  performance  programs  in  various 
scenarios  and  contribute  directly  to  the  above  priorities  by  achieving  affordable  programs 
that  are  successful  as  well  as  improving  overall  efficiency. 

Research  Methodology 

The  overall  goal  of  this  research  is  to  continue  our  efforts  to  forge  new  ground  on 
identifying  the  effects  of  interdependency  on  acquisition  and,  if  needed,  uncovering  early 
indicators  of  interdependency  risk  so  appropriate  governance  oversight  methods  can  then 
be  isolated.  Hence,  this  research  seeks  to  address  the  problem  that  there  is  little  insight  on 
the  effects  of  interdependencies  and  a  lack  of  tested  metrics  to  provide  early  indication  of 
the  acquisition  risks  of  interdependent  programs.  It  breaks  ground  in  the  area  of  (i)  studying 
non-linear  cascading  effects  in  the  context  of  a  network  of  MDAPs  consisting  of  some  not- 
so-successful  programs  (that  which  experiences  cost  growth)  as  compared  to  (ii)  the  study 
of  the  decision  mechanisms  of  successful  programs. 
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The  information  pertaining  to  acquisition  research  is  overwhelming  and  multifarious. 
It  appears  to  be  a  daunting  task  for  the  acquisition  researchers,  let  alone  the  program 
managers,  to  integrate  and  understand  the  vast  and  dynamic  data  in  a  coherent  way.  To 
define  the  interrelationship  among  the  MDAPs  from  a  network-centric  viewpoint  and  to 
identify  different  network  dependencies  within  the  domain  of  MDAPs,  the  following  set  of 
data  resources  are  useful: 

•  Monthly  DAES  reports  that  provide  an  early-warning  report  on  the  status  of 
some  program  features  such  as  cost,  schedule,  performance,  funding,  etc. 

•  SARs  that  summarize  the  latest  estimates  of  cost,  schedule,  and  technical 
status  to  be  reported  annually  in  conjunction  with  the  President’s  budget 

•  Program  Element  (PE)  documents  (called  PE  docs  or  R-docs)  that  are  used 
to  justify  congressional  budgeting  process 

•  Program  Objective  Memoranda  (POMs)  which  are  submitted  by  the 
components  (military  departments  and  DoD  agencies)  to  OSD  comptroller 

Next  we  describe  the  main  tenets  of  the  four  research  tasks  illustrated  in  Figure  1. 
Since  we  are  in  the  very  early  stages  of  this  project,  we  will  describe  our  proposed  research 
for  each  of  the  tasks  and  also  discuss  initial  progress  we  have  made  so  far. 


Task  1:  Network  Structure 
Formation  &  Analyrts 
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Figure  1.  Research  Goals 

Task  1:  Network  Structure  Formation  and  Analysis 

We  plan  to  address  the  following  two  questions  as  part  of  this  task: 

•  What  are  the  essential  features  of  the  network  that  reveal  the  joint  space 
dynamics? 

•  What  are  the  relative  priorities  associated  with  these  features  and  how  do 
they  affect  the  network  relationship? 
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Network  Structure  Formation.  From  our  previous  study,  we  identified  both 
successful  and  not-so-successful  programs  with  respect  to  performance  breaches.  For  the 
current  study,  we  plan  to  build  funding  networks  for  these  two  types  of  MDAPs.  We  will 
study  the  Program  Element  (PE)  accounts  of  these  programs  from  their  “Track  to  Budget” 
files  and  would  find  their  first-order  funding  neighbors.  This  process  would  enable  us  to 
define  the  network  topology  for  the  analysis  of  its  properties. 

Network  Structure  Analysis.  Network  theory  (Ahuja,  Magnanti,  &  Orlin,  1993; 
Albert,  Jeong,  &  Barabasi,  2000)  provides  useful  tools  to  calculate  and  understand 
quantities  or  measures  that  capture  significant  features  of  the  network  topology.  These 
measures  help  analyze  the  network  data  based  on  the  structure  of  the  network  and  also 
help  to  understand  how  those  properties  are  related  to  the  practical  issues  that  we  care 
about.  In  other  words,  network  theory  provides  a  rich  set  of  measures  and  metrics  that  can 
help  understand  what  the  network  data  may  tell.  A  key  metric  for  network  data  analysis  is 
various  types  of  centrality  measures.  Centrality  quantifies  how  important  are  the  nodes  (or 
edges)  in  a  networked  system.  There  are  a  wide  variety  of  mathematical  measures  of  node 
centrality  (Bonacich,  1987;  Borgatti,  2005;  Freeman,  Borgatti,  &  White,  1991)  that  focus  on 
different  concepts  and  definitions  of  what  it  means  to  be  central  in  a  network.  A  simple  but 
very  useful  example  is  the  measure  called  degree.  The  degree  of  a  node  in  a  network  is  the 
number  of  edges  attached  to  it. 

In  case  of  an  MDAP  funding  network,  degree-centrality  would  show  how  many 
funding  neighbors  a  particular  MDAP  has  and  how  it  could  be  related  to  the  performance  of 
the  program.  For  example,  having  many  funding  partners  incurs  more  risk  in  terms  of  being 
affected  by  the  cascading  consequences.  Many  of  the  standard  algorithms  for  the  study  of 
networks  are  already  available,  ready-made,  in  the  form  of  professional  network  analysis 
software  packages.  Some  of  the  software  packages  for  analysis  of  network  data  are  Paejk 
(http://vlado.fmf.uni-lj.si/pub/networks/Pajek/),  Netminer 
(http://www.netminer.com/index.php),  yEd 

(http://www.yworks.com/en/products_yed_about.html),  JUNG  (http://jung.sourceforge.net/), 
and  so  forth. 

State  of  the  program  in  our  decision-theoretic  DEC-MDP  model  captures  the  critical 
information  at  a  specific  point  in  time  that  will  support  the  decision-making  to  guarantee 
good  performance.  To  describe  the  state  space  and  to  identify  some  of  the  key  state 
features,  we  will  employ  an  appropriate  network  analysis  tool  for  the  MDAP  networks.  We 
plan  to  address  the  following  question:  What  are  the  network  properties  that  essentially 
contribute  towards  the  good/poor  performance  of  the  respective  MDAPs?  Our  goal  is  to 
measure  some  of  the  important  centrality  measures  for  the  network  and  correlate  it  with  the 
performance  of  the  node  (the  program).  Centrality  measures  help  us  to  determine  (i)  which 
nodes  are  important  in  the  network  and  (ii)  to  assess  their  importance  with  respect  to  their 
performance. 

We  plan  to  first  define  an  undirected  funding  network  for  a  chosen  MDAP.  We  will 
then  measure  the  following  network  centralities  for  5/10  years  time  span  for  all  MDAPs: 
degree,  betweenness,  closeness,  similarity,  local  clustering  coefficient,  and  so  forth.  We 
discuss  these  metrics  in  greater  detail  in  the  following  paragraphs.  We  also  plan  to  calculate 
the  performance  factor  for  5/10  years  time  span  for  all  MDAPs,  based  on  a  composite  metric 
(it  may  include  the  breach  factors,  %PAUC,  funding  delta,  and  so  forth  from  SAR  files).  This 
will  help  us  to  determine  how  each  of  the  centrality  measures  affects  the  performance  of  the 
programs  over  time. 
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The  above  methodology  will  enable  us  to  identify  additional  state  features  to  describe 
the  state  space  of  the  program  within  the  DEC-MDP  model.  The  following  is  the  list  of 
features  of  interest: 

•  Feature  1 :  Program  ID 

•  Feature  2:  Current  Year 

•  Feature  3:  Current  Month 

•  Feature  4:  Cost  (APB)  Status — for  nine  months,  starting  from  the  current 

month 

•  Feature  5:  Cost  (Contract)  Status — for  nine  months,  starting  from  the  current 
month 

•  Feature  6:  Schedule  (APB)  Status — for  nine  months,  starting  from  the  current 
month 

•  Feature  7:  Schedule  (Contract)  Status — for  nine  months,  starting  from  the 
current  month 

•  Feature  8:  Performance  (APB)  Status — for  nine  months,  starting  from  the 
current  month 

•  Feature  9:  Performance  (Contract)  Status — for  nine  months,  starting  from  the 
current  month 

•  Feature  10:  Funding  (APB)  Status — for  nine  months,  starting  from  the  current 
month 

•  Feature  11:  Funding  (Contract)  Status — for  nine  months,  starting  from  the 
current  month 

•  Feature  12:  Degree  Centrality 

•  Feature  13:  Closeness  Centrality 

•  Feature  14:  Betweenness  Centrality 

•  Feature  15:  Local  Clustering  Coefficient 

•  Feature  16:  Commodity  Type 

•  Feature  17:  Partner  Abandonment 

We  have  identified  Feature  1  through  Feature  1 1  to  be  useful  features  based  on  our 
past  work.  As  part  of  this  project,  we  propose  to  continue  studying  these  features  and 
introduce  more  network-centric  features  in  the  context  of  studying  the  role  of 
interdependencies  on  performance.  Features  12-17  capture  some  of  the  key  network¬ 
centric  features  for  the  MDAP  of  interest.  For  example,  Feature  12  (degree  centrality) 
measures  the  connectivity  of  a  program  with  other  programs.  A  higher  connectivity  might 
incur  higher  risk  because  of  its  sharing  of  funding  with  many  partners.  Feature  13 
(closeness  centrality)  measures  the  mean  distance  of  a  program  from  other  programs. 

These  centrality  measures  could  offer  better  understanding  about  the  propagation  speed  of 
the  cascading  effects.  Feature  14  (betweenness  centrality)  measures  the  importance  of  the 
program  that  may  reside  in  the  overlapping  region  of  more  than  one  sub-network  and  which 
is  able  to  control  the  flow  of  influence  among  different  sub-networks.  Feature  15  (local 
clustering  coefficient)  measures  the  formation  of  groups  among  the  member  nodes  and  it 
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may  be  related  to  the  degree  distribution  of  the  network.  For  example,  typically  nodes  with 
higher  degree  have  a  lower  local  clustering  coefficient  on  average.  Therefore,  a  node  with  a 
higher  local  clustering  coefficient  (and  lower  degree  distribution)  is  most  likely  prone  to  lower 
risk.  It  is  also  useful  to  identify  the  “structural  holes”  in  a  network.  If  two  neighbors  of  a  node 
are  not  themselves  neighbors,  then  we  say  that  there  is  a  “structural  hole”  existing  among 
them.  Identification  of  “structural  holes”  could  be  useful  to  analyze  the  propagation  of 
cascading  risks.  As  part  of  this  task,  we  will  study  the  usefulness  of  these  features  and  also 
identify  other  new  ones. 

Initial  Results  on  Task  1:  Network  Structure  Formation 

We  define  a  funding  network  of  an  MDAP  using  the  PEs  that  funded  the  MDAP’s 
RDT&E  efforts.  PE  is  the  code  number  assigned  by  the  comptroller.  Since  PEs  fund  multiple 
MDAPs,  programs  that  share  a  common  PE  monitor  could  be  isolated.  Procurement  PEs 
were  not  considered  for  defining  funding  networks  since  the  RDT&E  interdependencies 
were  the  most  critical  to  program  performance.  The  funding  network  and  the  associated  R- 
docs  allowed  us  to  do  a  detailed  study  of  the  performance  of  the  member  nodes  and  to 
understand  the  cascading  effects  the  funding  network  of  the  three  MDAPs  named  MDAP_A, 
MDAP_B  and  MDAP_C.  The  original  names  of  these  MDAPs  have  been  removed  to  retain 
the  confidentiality  of  the  programs. 

Examination  of  the  DAES  reports  and  R-docs  from  the  years  2006-201 1  related  to 
these  MDAPs  shows  that  MDAP_A  and  MDAP_B  experience  frequent  performance 
breaches  while  MDAP_C  appears  to  be  performing  as  expected.  We  have  built  an  evolving 
funding  network  of  these  three  MDAPs  based  on  the  common  PE  accounts  that  they  share 
with  other  MDAPs,  such  as  MDAP_D-I.  The  relationship  between  the  PE  accounts  and  the 
MDAPs,  extracted  from  the  PE  docs,  is  represented  as  bipartite  networks.  Figure  2  shows 
how  the  funding  relationship  of  these  three  MDAPs  and  their  neighbors  change  from  2006  to 
201 1 .  Since  the  PE  docs  for  the  year  2008  were  unavailable,  we  couldn’t  show  the  funding 
network  for  that  year. 
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Figure  2.  Evolving  Funding  Network  of  MDAP_A,  MDAP_B,  and  MDAP_C 


From  these  bipartite  networks,  we  notice  that  MDAP_A  and  MDAP_B  share  only  one 
PE  account  (PE  1),  while  MDAP_C  shares  multiple  PE  accounts  (PE  1-4).  It  indicates  that 
MDAP_C  is  prone  to  more  inter-dependency  risks. 

Next  we  plan  to  measure  the  weight  of  the  links  between  the  PE  account  and  the 
respective  MDAPs  based  on  the  funding  distribution  as  captured  in  the  PE  docs.  This 
measurement  can  be  obtained  by  comparing  the  POM  and  SARS  data.  The  former 
describes  what  the  PM  says  the  program  requires  and  the  latter  is  what  the  program  actually 
got.  This  comparison  will  give  us  a  better  understanding  of  the  dependency  of  MDAPs  on 
the  associated  PEs  and  the  effect  of  expected  and  actual  budget  allocations  on  performance 
breaches.  We  will  use  these  link  weights  as  state  features  for  the  respective  programs. 

Task  2:  Automated  Data  Extraction  and  Text  Analysis 

We  plan  to  address  the  following  two  questions  as  part  of  this  task: 

•  What  are  the  local  issues  that  lead  toward  breach  or  near-breach  situations? 

•  How  often  and  why  do  the  local  mitigation  efforts  fail  to  improve  the 
performance? 

•  How  do  we  identify  the  non-local  issues  that  result  from  the 
interdependencies? 

•  How  do  we  determine  the  cascading  effect  through  the  network? 

We  plan  to  approach  Task  2  from  two  perspectives:  Local  perspective  where  the 
analyses  are  based  solely  on  the  individual  program’s  own  data;  and  Non-Local  perspective 
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where  the  analyses  are  based  on  the  data  of  MDAPs  existing  in  the  joint  space  of  the 
individual  program.  Lessons  learned  from  these  analyses  should  enable  the  stakeholders  to 
take  appropriate  measures  to  improve  the  performance  of  the  programs.  Our  objective  in 
this  task  is  to  narrow  down  the  wealth  of  data  present  in  the  DAES  reports  in  order  to 
capture  useful  knowledge  about  the  status  of  individual  MDAPs  and  the  MDAPs  in  their 
network.  This  will  be  achieved  as  follows: 

Automatic  Data  Extraction.  The  aim  of  this  subtask  is  to  bring  the  content  of  DAES 
reports,  currently  as  Microsoft  PowerPoint  files,  Adobe  Acrobat  PDF  files,  and  Word 
documents,  into  a  form  suitable  to  further  analysis.  We  will  mainly  focus  on  the  program 
status  and  issue  summary.  First,  bottom-up  (pixel  to  block)  image  segmentation  will  be  used 
in  order  to  extract  the  layout  of  the  document  (O’Gorman,  1993;  O’Gorman  &  Kasturi,  1997; 
Salleb  &  Hocini,  1996).  It  appears  from  the  DAES  reports  that  the  part  that  requires  further 
extraction  is  the  program  status  matrix  for  the  following  items:  Cost,  Schedule,  Performance, 
and  Funding.  The  status  of  each  of  these  items  is  given  for  APB  and  Contract.  The  status  is 
a  colored  circle  indicating  three  possible  states:  meet  all  contracts  (green);  resolvable 
contracts  (yellow),  and  cannot  meet  all  contracts  (red).  The  status  is  given  for  the  current 
month,  past  three  months,  along  with  a  forecast  for  the  upcoming  nine  months. 

Once  we  extract  the  different  components  in  the  document  though  image 
segmentation  using  bounding  boxes,  nearest  neighbors,  linear  regression  (O’Gorman,  1993; 
Salleb  &  Hocini,  1996),  we  will  translate  the  program  status  matrix  into  an  integer-valued 
matrix,  where  green  will  be  represented  by  1 ,  yellow  by  0,  and  red  by  -1 .  An  example  of 
such  a  representation  is  presented  in  Figure  3.  We  will  also  parse  DAES  files  to  extract  all 
the  words  used  in  the  program  status  and  issue  summary.  We  will  use  Java  text  extraction 
libraries  that  have  proven  to  be  powerful.  Hence,  a  report  will  be  defined  by  the  following 
components:  MDAP  name,  Month,  Year,  status  matrix,  and  the  extracted  text  from  the 
program  status  description  and  from  the  issue  summary. 
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Figure  3.  Translation  of  Data  to  Integer-Valued  Matrix 


Advanced  Text  Analysis.  Text  from  the  program  status  and  issue  summary  will  be 
used  to  assess  the  health  or  status  of  the  MDAP.  We  will  extensively  use  word  clouds  in 
order  to  visualize  the  status  of  the  MDAP  as  described  in  the  corresponding  text  (see 
example  of  word  cloud  in  Figure  4),  while  word  clouds  will  provide  a  nice  visualization  tool 
that  provides  a  general  idea  on  the  contents  of  the  documents. 
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Besides  visualization,  we  will  also  extract  n-grams  (sequence  of  words)  and 
concepts  or  topics  from  the  text  in  order  to  identify  the  issues  that  an  MDAP  is  going 
through.  For  this  purpose,  we  will  use  Topic  Modeling  to  discover  the  topics  discussed  in  the 
corpus.  The  intuition  behind  topic  modeling  is  that  when  program  managers  prepare  to  write 
their  monthly  reports,  they  first  have  in  mind  a  set  of  topics  to  address.  They  fill  in  the  DAES 
using  words  associated  with  the  different  topics.  Topic  modeling  identifies  which  words  have 
the  greatest  probability  of  occurring  together,  and  posits  an  abstract  topic  that  conditions 
these  probabilities.  After  generating  the  topic  models  for  the  MDAP  documents,  each 
document  can  be  represented  as  a  subset  of  the  total  topics,  each  in  a  proportion 
dependent  on  the  content  words.  For  instance,  a  report  can  belong  to  the  topic  “delay  in 
schedule”  and  also  to  a  topic  “gap  in  funding.” 

To  preprocess  the  documents,  we  will  strip  all  the  non-content  words,  and  keep  only 
the  free  text.  Words  and  characters  that  are  removed  include  section  and  field  names, 
person  names,  punctuation,  digits,  and  stop-words.  A  topic  model  consists  of  a  probability 
distribution  over  topics,  and  then  for  each  topic,  the  probability  of  each  word  in  the 
vocabulary.  The  parameters  behind  the  probability  distributions  are  treated  as  latent 
variables.  By  analyzing  a  set  of  observations  (words  in  the  documents),  it  is  possible  to 
recover  the  latent  structure  of  the  generative  model.  The  particular  model  we  use  is  based 
on  Latent  Dirichlet  Allocation  (LDA;  Blei,  Ng,  &  Jordan,  2003)  with  Gibbs  Sampling.  For  the 
experiment,  we  will  use  the  Stanford  Topic  Modeling  tool  kit 

(http://nlp.stanford.edU/software/tmt/tmt-0.4/),  a  machine  learning  toolkit  for  natural  language 
processing  tasks. 

Initial  Results  on  Task  2:  Automated  Data  Extraction 

As  discussed  previously,  DAES  reports  include  information  of  program  performance 
in  the  form  of  text  and  image.  Our  current  focus  is  to  understand  the  textual  descriptions  in 
the  reports.  The  “Issue  summary”  section  in  the  report  illustrates  the  local  issues,  if  any,  and 
possible  actions  to  resolve  them.  We  prepare  the  input  file  for  the  topic  modeling  tool  by 
manually  copying  this  information  as  records  into  a  csv  (comma  separated  value)  file. 
Specifically,  we  created  two  input  files,  one  with  set  of  Issues  (problems  encountered  by 
MDAPs  as  reported  in  the  DAES)  and  the  other  with  set  of  Actions  (the  tangible  actions 
proposed  by  the  MDAP  program  manager  to  alleviate  the  Issues).  As  described  above,  we 
preprocess  the  reports  by  stripping  the  non-content  words,  and  only  keep  the  free  text. 
Words  and  characters  that  are  removed  include  section  and  field  names,  person  names, 
punctuation,  digits,  and  stop-words. 

We  first  train  a  classifier  to  automatically  identify  the  Issues  identified  in  the  DAES 
reports.  Using  an  input  file  for  the  program  MDAP_A  from  the  previous  section  with  few  (15) 
records  of  its  issues  from  a  single  year,  we  trained  a  model  that  will  classify  contents  into 
issue-related  topics.  The  results  were  not  informative  as  the  data  was  small,  and  so  we 
extended  the  input  to  include  issues  of  all  the  reports  of  MDAP_A  across  the  years.  The 
increased  data  set  resulted  in  words  like  schedule,  Funding,  Launch,  ground  site,  and 
control  to  be  the  top  words  in  individual  topic  list.  Examination  of  the  tool  for  consistent 
results  is  important,  and  this  technically  indicates  the  convergence  of  the  model. 
Convergence  is  dependent  on  the  number  of  iterations  the  model  is  executed,  which  in  turn 
is  dependent  on  the  data  size.  For  a  data  size  of  100  plus  records,  convergence  occurred  at 
around  800  iterations.  We  tested  our  trained  model  on  a  few  (30)  records  of  the  same 
MDAP_A  program.  Test  results  indicate  the  proportion  of  relevance  of  the  record  to  each  of 
the  topics.  In  Figure  5,  we  describe  an  example  record  and  the  proportion  to  which  the 
record  is  relevant  to  the  five  topics  identified  by  the  model:  Schedule,  Funding,  Launch, 
Ground  site,  Cost  Control.  As  shown,  this  record  has  a  high  proportion  of  the  topic  Funding. 
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Figure  5.  Example  Topic  Distribution  for  an  “Issue-Related”  Record 
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Figure  6.  Determining  Optimal  k  Value  for  an  “Issue-Related”  Topic 

The  number  of  topics  is  another  important  parameter  for  topic  modeling.  Initial  results 
included  five  topics  but  finding  optimum  value  for  the  number  of  topics  will  provide  better 
results  (Griffiths  &  Steyvers,  2004)  in  the  sense  that  topics  will  be  of  finer  granularity  and 
hence  more  specific  and  relevant.  For  this  we  trained  the  model  several  times  and  recorded 
perplexity.  Perplexity  is  a  measure  of  the  quality  of  the  model  learned  by  LDA  in  predicting 
future  data  from  the  same  distribution  as  the  data  used  to  train  the  model.  Lower  perplexity 
value  indicates  a  stable  model.  An  experiment  with  the  different  number  of  topics,  as  shown 
in  Figure  6,  signifies  that  a  k  value  of  20  or  more  is  the  best  for  our  experimental  data. 

Our  next  steps  in  the  task  will  involve  the  following: 

•  Automate  the  preparation  of  the  input  file  using  PERL,  a  scripting  language. 

•  Expand  the  input  data  set  to  include  reports  of  all  the  programs  across  the 
years  and  train  the  model  with  this  data. 
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•  Explore  parameters  of  the  LDA  model  to  fine-tune  the  results  such  that  the 
top  set  of  words  in  a  topic  list  is  explanatory  of  that  topic. 

•  Frame  a  phrase  by  analyzing  the  word  list,  for  example,  Hardware  issue  for 
better  understanding  and  to  support  further  analysis. 

•  Perform  a  similar  topic  extraction  of  “Action”  related  data. 

•  Scale  the  analysis  to  all  MDAP  programs. 

•  Use  the  extracted  information  to  populate  the  Markov  Decision  Process  in 
Task  4. 

•  Apply  these  topic  extraction  techniques  to  POM  documents  and  compare  it  to 
the  information  in  the  SARS  documents  as  discussed  in  Task  1. 

Task  3:  Local/Non-Local  Issue  Analysis 

As  part  of  automating  the  identification  and  analysis  of  local  and  non-local  issues  that 
lead  to  performance  breaches,  we  will  first  evaluate  the  monthly  mitigation  forecasting  for 
the  problems  from  the  DAES  reports.  We  hypothesize  that  frequent  forecasting  failure  along 
with  sustaining/recurring  breaches  would  require  issue  analysis.  We  plan  to  analyze  the 
automatically  extracted  issues  (Task  2)  to  reveal  the  presence  or  absence  of  local  issues  to 
explain  the  erroneous  forecasting.  If  no  significant  issue  can  be  found  to  explain  the  frequent 
forecasting  failure,  then  we  claim  that  either  DAES  reports  do  not  capture  the  local  reasons, 
or  some  non-local  reasons  are  responsible  for  the  poor  performance.  We  will  then  analyze 
the  local  issues  of  the  neighbors  in  the  funding  network  to  determine  if  there  is  any  non-local 
issue  that  possibly  could  have  propagated  through  the  network  leading  to  performance 
breaches.  This  is  work  that  we  will  pursue  after  we  make  progress  on  Task  2. 

Task  4:  Formulate  a  Decision-Theoretic  Model  That  Harnesses  Decentralized-Markov 
Decision  Process  (DEC-MDP)  Formalism 

The  questions  to  be  addressed  by  this  task  are  as  follows: 

•  What  are  the  essential  characteristics  of  the  MDAP  network  that  justify  a 
DEC-MDP  model? 

•  How  to  model  the  MDAP  network  as  a  decentralized  system? 

•  What  are  the  key  challenges  in  the  design  of  the  DEC-MDP? 

•  What  essential  features  should  the  DEC-MDP  model  incorporate  for  better 
predictability? 

In  this  work,  decision-making  in  an  MDAP  network  is  viewed  as  a  multiagent 
sequential  decision  problem  because  the  utility  gained  by  each  agent  depends  on  a 
sequence  of  actions  over  time.  Our  goal  is  to  determine  the  behavior  of  the  decision-makers 
(agents)  that  best  balances  the  risks  and  rewards  while  acting  in  an  uncertain  environment 
with  stochastic  actions. 

Each  agent  will  make  its  individual  decisions  in  an  environment  where  the  state 
space  is  not  fully  observable,  meaning,  that  the  nodes  in  the  network  (the  programs)  do  not 
exactly  know  in  which  state  they  are  in  at  any  particular  instant  because  they  do  not  have 
complete  information  about  their  neighbors.  With  the  partial  state  information,  the  individual 
agents  aim  to  optimize  the  joint  reward  function.  This  class  of  problems  is  modeled  as 
decentralized  partially  observable  MDP  (DEC-POMDP)  in  literature  (Bernstein  et  al. ,  2002) 
where  at  each  step  when  an  agent  takes  an  action,  a  state  transition  occurs,  and  the  agent 
receives  a  local  observation.  Following  this,  the  environment  generates  a  global  reward  that 
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depends  on  the  set  of  actions  taken  by  all  the  agents.  A  necessary  condition  for  stable 
equilibrium  among  agents  in  a  multiagent  system  is  that  each  agent  plays  a  best-response 
to  the  strategy  of  every  other  agent:  this  is  called  a  Nash  Equilibrium.  In  our  previous  work 
(Cheng,  Raja,  &  Lesser,  2012)  we  make  the  DEC-POMDP  problem  for  a  tornado  tracking 
tractable  by  approximating  the  DEC-POMDP  with  a  stochastic  DEC-MDP  model  and  using  a 
factored  reward  function  to  define  a  Nash  Equilibrium  instead  of  the  global  reward  function. 
We  apply  this  technique  to  the  MDAP  domain.  We  define  the  reward  function  of  this  model 
to  be  composed  of  two  different  components:  local  reward  function  and  global  reward 
function.  The  local  reward  functions  are  dependent  only  on  the  individual  agent’s  actions, 
while  the  global  reward  function  depends  on  the  action  of  all  agents.  We  make  this  a 
stochastic  DEC-MDP  by  defining  a  solution  as  a  stochastic  policy  for  each  agent.  A 
stochastic  policy  of  an  agent  /  is  denoted  by  rtj(s)  e  PD  (A,),  where  PD  (Aj),  is  the  set  of 
probability  distributions  over  actions  Aj.  Stochastic  policies  can  cope  with  the  uncertainty  of 
observation  and  perform  better  than  deterministic  policies  in  a  partial  observable 
environment.  We  plan  to  apply  these  modeling  techniques  we  have  developed  for  another 
complex  multiagent  domain  (tornado  tracking)  to  the  MDAP  domain. 

Conclusions  and  Future  Work 

Our  multi-year  research  goal  is  to  gain  a  deeper  understanding  of  interdependencies 
among  MDAPs  by  examining  the  various  information  sources,  SARS,  DAES,  POMS,  and  R- 
docs.  This  would  involve  establishing  a  statistically  significant  correlation  between  the  state 
of  MDAP  network  dependencies  and  their  consequences.  Our  previous  work  in  this  area 
involved  manual  analysis  of  DAES  and  SARS  data  belonging  to  a  small  network  MDAPs  to 
determine  the  local  and  non-local  issues  that  affect  MDAP  performance.  As  a  consequence 
of  this  work,  we  recognized  the  need  to  analyze  the  data  from  the  entire  set  of  MDAPs  in 
batch  form  to  be  able  to  build  good  decision  models  for  “what-if  analysis.  The  volume  and 
complexity  of  the  data  has  led  to  our  current  research  tasks  that  involve  automating 
methods  for  data  extraction,  network  analysis,  and  decision  model  construction  for 
successful  and  not-so-successful  MDAPs.  In  this  paper,  for  each  task,  we  describe  our 
proposed  work  and  initial  results.  Our  hope  is  that  as  a  consequence  of  this  work,  we  will  be 
able  to  (1)  extract  the  link  characteristics  between  MDAPs;  (2)  examine  and  compare  the 
funding  network  structure  characteristics  of  interdependent  regions  belonging  to  successful 
and  not-so-successful  MDAPs  to  augment  our  current  work  in  “what-if”  analyses;  (3) 
automate  the  data  extraction  and  analysis  process  by  leveraging  algorithms  for  decision 
support  as  well  as  image  and  text  analysis;  and  (4)  continue  to  identify  the  challenges  in 
acquiring  the  data  from  the  government  and  program  managers. 
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Abstract 

DoD  acquisition  is  an  extremely  complex  system,  comprised  of  myriad  stakeholders, 
processes,  people,  activities,  and  organizational  structures.  Processes  within  this  complex 
system  are  encumbered  by  the  continuous  development  of  large  amounts  of  unstructured 
and  unformatted  acquisition  program  data,  difficult  to  aggregate  across  the  “enterprise.”  Yet 
acquisition  analysts  and  decision-makers  must  analyze  all  types  and  spectrums  of  the 
available  data  to  obtain  a  complete  and  comprehensible  picture.  This  can  be  a  daunting  task. 
We  have  applied  a  data-driven  automation  system  and  methodology,  namely,  lexical  link 
analysis  (LLA),  to  facilitate  acquisition  researchers  and  decision-makers  to  recognize 
important  connections  (concepts)  that  form  patterns  derived  from  dynamic,  ongoing  data 
collection,  analysis,  and  decision  making.  LLA  technology  and  methodology  is  used  to 
uncover  and  display  relationships  among  competing  programs  and  Navy-driven 
requirements.  In  the  past  year,  we  tested  our  method  using  samples  of  acquisition  data  for 
visualization  and  validity.  LLA  successfully  discovered  statistically  significant  correlations,  and 
automatically  extracted  lexical  links,  thus  improving  acquisition  professionals’  knowledge  of 
their  data.  This  might  have  otherwise  required  expensive  manpower  to  perform.  We  also 
developed  LLA  into  a  web  service  via  several  use  cases  for  large-scale  LLA  applications.  In 
this  paper,  we  show  how  to  apply  the  LLA  web  service  to  the  Acquisition  Visibility  Portal, 
which  is  a  critical  tool  to  provide  the  DoD-wide  acquisition  community  with  authoritative  and 
accurate  data  services.  The  resulting  methodology  could  reduce  the  workload  of  decision¬ 
makers  and  achieve  improved  purchasing  decisions,  serving  to  improve  the  long-term 
success  of  acquisition  strategies. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  153- 


Introduction 

Acquisition  research  has  increased  in  component,  organizational,  technical,  and 
management  complexity.  It  is  difficult  for  acquisition  professionals  to  remain  continuously 
aware  of  their  decision-making  domains  because  information  is  overwhelming  and  dynamic. 
According  to  the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  for  Joint  Capabilities 
Integration  and  Development  System  (JCIDS;  CJCS,  2009),  there  are  three  key  processes 
in  the  DoD  that  must  work  in  concert  to  deliver  the  capabilities  required  by  the  warfighters: 
the  requirements  process;  the  acquisition  process;  and  the  Planning,  Programming,  Budget, 
and  Execution  (PPBE)  process. 

Each  process  produces  a  large  amount  of  data  in  an  unstructured  manner;  for 
example,  the  warfighters’  requirements  are  documented  in  Universal  Joint  Task  Lists 
(UJTLs),  Joint  Capability  Areas  (JCAs),  and  Urgent  Need  Statements  (UNSs).  These 
requirements  are  processed  in  the  JCIDS  to  become  projects  and  programs,  which  should 
result  in  products  such  as  weapon  systems  that  meet  the  warfighters’  needs.  Program  data 
are  stored  in  the  Defense  Acquisition  System  (DAS).  Programs  are  divided  into  Major  DoD 
Acquisition  Programs  (MDAPs),  Acquisition  Category  II  (ACATII),  and  so  forth.  Program 
Elements  (PEs)  are  the  documents  used  to  fund  programs  yearly  through  the  congressional 
budget  justification  process.  Data  is  too  voluminous,  too  unformatted,  and  too  unstructured 
to  be  easily  digested  and  understood — even  by  a  team  of  experienced  acquisition 
professionals. 

On  a  conceptual  level,  our  first  question  is  as  follows:  How  can  the  information  that 
emerges  from  the  acquisition  process  be  used  to  produce  overall  awareness  of  the  fit 
between  programs,  projects,  systems,  and  of  the  needs  for  which  they  were  intended? 


In  precise  terms,  we  observed  that  there  were  three  important  processes  that 
seemed  fundamentally  disconnected.  Specifically,  they  were  the  congressional  budgeting 
justification  process  (such  as  information  contained  within  the  PEs),  the  acquisition  process 
(such  as  information  in  the  MDAP  and  ACATII),  and  the  warfighters’  requirements  (such  as 
information  in  UNSs  and  in  UJTLs),  as  shown  in  Figure  1.  Yet,  these  were  not  analyzed  and 
compared  together  in  a  dynamic,  holistic  methodology  that  could  keep  pace  with  changes 
and  reflect  patterns  of  relationships.  In  the  past  three  years,  we  employed  the  lexical  link 
analysis  (LLA)  automation  methodology  to  analyze  the  data  in  three  areas,  illustrated  in 
Figure  1 . 
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Figure  1.  Determining  Business  Processes  Links  From  Requirements  to 
DoD  Budget  Justification  to  Final  Products 
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In  the  past,  we  have  explored  how  analytic  and  visualization  tools  such  as  LLA  can 
help  detect  data  inconsistency  and  gaps  (bad  data;  see  Research  Status  section).  We  have 
further  systematically  improved  our  understanding  of  the  quality  of  the  data  by  automatically 
discovering  new  patterns  that  were  previously  unknown,  and  identified  data  dependencies 
that  might  be  indicators  for  program  or  investment  performances.  However,  much  more  work 
is  needed  in  this  area  as  well  as  continued  in-depth  analysis  performed  at  the  different 
levels  of  the  Acquisition  Visibility  Portal  (AVP).  AVP  is  a  critical  tool  that  provides  the  DoD- 
wide  acquisition  community  with  authoritative  and  accurate  data  services  via  interfaces  to 
DTIC  and  DAMIR  for  programs  (e.g.,  MDAPs,  ACATIIs)  with  milestones,  costs,  schedules 
and  performance  data,  Selected  Acquisition  Reports  (SARs),  and  Acquisition  Strategy 
Reports  (ASRs),  among  others. 

We  seek  to  show  how  LLA  can  be  adapted  to  the  AVP’s  ongoing  requirements  and 
continuous  improvement  of  DoD  data  quality  and  decision-making. 

Methodology 

Overview  of  Lexical  Link  Analysis 

As  in  military  operations,  where  the  term  situational  awareness  was  coined,  we  note 
that  our  efforts  can  inform  awareness  of  analyzed  data  in  a  unique  way  that  helps  improve  a 
decision-maker’s  understanding  or  awareness  of  its  content.  We  therefore  define  awareness 
as  the  cognitive  interface  between  decision-makers  and  a  complex  system,  expressed  in  a 
range  of  terms  or  “features,”  or  specific  vocabulary  or  “lexicon,”  to  describe  the  attributes 
and  surrounding  environment  of  the  system.  Specifically,  LLA  is  a  form  of  text  mining  in 
which  word  meanings  represented  in  lexical  terms  (e.g.,  word  pairs)  can  be  represented  as  if 
they  are  in  a  community  of  a  word  network. 

Link  analysis  “discovers”  and  displays  a  network  of  word  pairs.  These  word  pair 
networks  are  characterized  by  one-,  two-,  or  three-word  themes.  The  weight  of  each  theme 
is  determined  by  its  frequency  of  occurrence.  Figure  2  shows  a  visualization  of  common 
lexical  links  shared  between  Systems  1  and  2,  shown  in  the  red  box.  Unlinked,  outer  vectors 
(outside  the  red  box)  indicate  unique  system  features.  For  example,  Figure  3  shows  the 
information  from  three  categories  that  can  be  compared,  and  Figure  4  shows  the  information 
from  two  time  periods  that  can  be  compared. 

Each  node,  or  word  hub,  represents  a  system  feature,  and  each  color  refers  to  the 
collection  of  lexicon  links  (features)  that  describes  a  concept  or  theme.  The  overlapping  area 
nodes  are  lexical  links.  What  is  unique  here  is  that  LLA  constructs  these  linkages  via 
intelligent  agent  technology  using  social  network  grouping  methods. 

The  closeness  of  the  systems  in  comparison  can  be  visually  examined  or  examined 
using  the  Quadratic  Assignment  Procedure  (QAP;  Hubert  &  Schultz,  1976;  e.g.,  in  UCINET; 
Borgatti,  Everett,  &  Freeman,  2002)  to  compute  the  correlation  and  analyze  the  structural 
differences  in  the  two  systems,  as  shown  in  Figure  5. 
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Figure  2.  Comparing  Two  Systems  Using  LLA 


Figure  3.  Comparing  Three  Categories  Using  LLA 
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Figure  4.  Comparing  Two  Time  Periods 
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0.020  0.020  0.020  0.020  0.020  0.020  0.020  0.000 


qap  statistics  saved  as  datafile  qap  correlation  Results 


Figure  5.  QAP  Correlation  via  UCINET 


Figure  6  shows  a  visualization  of  LLA  with  connected  keywords  or  concepts  as 
groups  or  themes.  Words  are  linked  as  word  pairs  that  appear  next  to  each  other  in  the 
original  documents.  Different  colors  indicate  different  clusters  of  word  groups.  They  were 
produced  using  a  link  analysis  method — a  social  network  grouping  method  (Girvan  et  al., 
2001)  where  words  are  connected,  as  shown  in  a  single  color,  as  if  they  are  in  a  social 
community.  A  “hub”  is  formed  around  a  word  centered  or  connected  with  a  list  of  other 
words  (“fan-out”  words)  centered  on  other  hub  words.  For  instance,  Figure  7  shows  a 
detailed  view  of  a  theme  or  word  group  in  Figure  6:  the  words  “analysis,  research,  approach 
are  connected  and  centered  around  other  related  words.  We  use  three  words  such  as 
“analysis,  research,  approach”  to  label  a  group. 
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Figure  6.  Word  and  Term  of  Themes  Discovered  and  Shown  in  Colored  Groups 


Figure  7.  A  Detailed  View  of  a  Theme  or  Word  Group  From  Figure  6 

The  detailed  steps  of  LLA  processing  include  applying  collaborative  learning  agents 
(CLAs)  and  generating  visualizations,  including  a  lexical  network  visualization  via  AutoMap 
(2009),  radar  visualization,  and  matrix  visualization  (Zhao,  Gallup,  &  MacKinnon,  2010).  The 
following  are  the  steps  for  performing  an  LLA: 

•  Read  each  set  of  documents. 
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•  Select  feature-like  word  pairs. 

•  Apply  a  social  network  community  finding  algorithm  (e.g.,  Newman  grouping 
method;  Girvan  et  al.,  2001)  to  group  the  word  pairs  into  themes.  A  theme 
includes  a  collection  of  lexical  word  pairs  connected  to  each  other. 

•  Compute  a  “weight”  for  a  theme  for  the  information  of  a  time  period,  that  is, 
how  many  word  pairs  belong  to  a  theme  for  that  time  period  and  for  all  time 
periods. 

•  Sort  theme  weights  by  time,  and  study  the  distributions  of  these  themes  by 
time. 

Business  Problems  That  LLA  Addresses 

General  areas  that  LLA  usually  informs  are  the  following: 

•  Discovering  themes  and  topics  in  the  unstructured  documents  and  sorting  the 
importance  of  the  themes 

•  Discovering  social  and  semantic  networks  of  organizations  that  were 
involved,  comparing  the  two  networks  to  obtain  insights  to  answer  the 
following  questions: 

o  Demonstrating  what  were  the  organizations  involved  in  the  important 
themes 

o  Illustrating  how  semantic  networks  might  suggest  improved  potential 
collaboration  when  compared  to  social  networks 

Social  and  Semantic  Networks  Analysis 

Current  research  of  social  network  analysis  mostly  focuses  on  people  or 
organizations  of  direct  associations,  regardless  of  the  contents  linked.  The  so-called  study  of 
centrality  (Girvan,  2002;  Feldman,  2007)  has  been  a  focal  point  for  the  social  network 
structure  study.  Finding  the  centrality  of  a  network  lends  insight  into  the  various  roles  and 
groupings  such  as  the  connectors  (e.g.,  mavens,  leaders,  bridges,  isolated  nodes),  the 
clusters  (and  who  is  in  them),  the  network  core,  and  its  periphery.  We  have  been  working 
toward  two  areas  of  innovations  in  the  network  analysis: 

•  Extracting  social  networks  based  on  the  entity  extraction 

•  Extracting  semantic  networks  based  on  the  contents  and  word  pairs  using 
LLA 

•  Applying  characteristics  and  centrality  measures  from  the  semantic  networks 
and  social  networks  to  predict  latent  properties  such  as  emerging  leadership, 
for  example,  emerging  techniques  that  might  dominate,  in  the  social 
networks.  These  characteristics  are  further  categorized  into  themes  and  time- 
lined  trends  for  informed  prediction  of  future  events. 

Implementation  Details 

In  the  past  year,  we  continued  our  efforts  at  the  Naval  Postgraduate  School  (NPS)  by 
using  CLAs  (Quantum  Intelligence  [Ql],  2009)  and  expanded  to  other  tools,  including 
AutoMap  (Center  for  Computational  Analysis  of  Social  and  Organizational  Systems 
[CASOS],  2009)  for  improved  visualizations.  Results  from  these  efforts  arose  from 
leveraging  intelligent  agent  technology  via  an  educational  license  with  Quantum  Intelligence, 
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Inc.  CLA  is  a  computer-based  learning  agent,  or  agent  collaboration,  capable  of  ingesting 
and  processing  data  sources. 

We  have  been  generating  visualizations  including  a  lexical  network  visualization 
using  various  open  source  tools.  We  began  by  using  the  Organizational  Risk  Assessment 
(ORA;  CASOS,  2009)  tool  and  expanded  to  other  tools.  For  example,  in  the  past  year,  we 
developed  3-D  network  views  using  Pajek  (201 1 )  and  X3D  (201 1 ).  We  also  developed  our 
visualizations  Radar  view  and  Match  view  (Zhao  et  al.,  2010). 

LLA  uses  a  computer-based  learning  agent  called  Collaborative  Learning  Agents 
(CLA;  Ql,  2009)  to  employ  an  unsupervised  learning  process  that  separates  patterns  and 
anomalies.  CLA  is  a  computer-based  learning  agent,  or  agent  collaboration,  capable  of 
ingesting  and  processing  data  sources,  leveraged  via  an  educational  license  with  Ouantum 
Intelligence,  Inc.  The  unsupervised  agent  learning  is  implemented  by  indexing  each  set  of 
documents  separately  and  in  parallel  using  multiple  learning  agents.  Multiple  agents  can 
work  collaboratively  and  in  parallel.  We  set  up  a  cluster  utilizing  Linux  servers  in  the  NPS 
High  Performance  Computing  Center  (HPC)  to  handle  the  large-scale  data  and  secure 
environment  in  the  NPS  Secure  Technology  Battle  Laboratory  (STBL). 

Relations  to  Other  Methods 

The  LLA  approach  is  more  properly  related  to  Latent  Semantic  Analysis  (LSA; 
Dumais,  Furnas,  Landauer,  Deerwester,  &  Harshman,  1988)  and  Probabilistic  Latent 
Semantic  Analysis  (PLSA).  In  the  LSA  approach,  a  term-document  matrix  is  the  starting 
point  for  analysis.  The  elements  of  the  term-document  or  feature-object  (term  as  feature  and 
document  as  object)  matrix  are  the  occurrences  of  each  word  in  a  particular  document,  that 
is,  A  =  [a.ij],  where  ay  denotes  the  frequency  in  which  term  j  occurs  in  document  /.  The  term- 
document  matrix  is  usually  sparse.  LSA  uses  singular  value  decomposition  (SVD)  to  reduce 
the  dimensionality  of  the  term-document  matrix.  SVD  cannot  be  applied  to  the  cases  where 
the  vocabulary  (the  unique  number  of  terms)  in  the  document  collection  is  large.  LSA  has 
been  widely  used  to  improve  information  indexing,  search/retrieval,  and  text  categorization. 

A  recent  development  related  to  this  method  is  called  latent  Dirichlet  allocation  (LDA; 
Blei,  Ng,  &  Jordan,  2003),  which  is  a  generative  probabilistic  model  of  a  corpus.  In  LDA,  a 
document  is  considered  to  be  composed  of  a  collection  of  words — a  “bag  of  words,”  where 
word  order  and  grammar  are  not  considered  important.  The  basic  idea  is  that  documents 
are  represented  as  random  mixtures  over  latent  topics,  where  each  topic  is  characterized  by 
a  statistical  distribution  (Dirichlet  distribution)  over  the  corpus.  Our  theme  generation  from 
LLA  is  different  than  LDA,  in  which  a  collection  of  lexical  terms  are  connected  to  each  other 
semantically,  as  if  they  are  in  a  social  community,  and  social  network  grouping  methods  are 
used  to  group  the  words,  and  unlike  LSA,  our  method  is  easily  scaled  to  analyze  a  large 
vocabulary  and  is  generalizable  to  any  sequential  data. 

Anticipated  Benefits 

Our  LLA  method  provides  the  solutions  to  meet  the  critical  needs  of  the  acquisition 
research.  The  key  advantage  is  to  provide  an  innovative  near  real-time  self-awareness 
system  to  transfer  diversified  data  services  into  strategic  decision-making  knowledge, 
specifically  through  the  following: 

•  Automation:  High  correlation  of  LLA  results — with  the  link  analysis  done  by 
human  analysts — makes  it  possible  to  save  human  power  and  improve 
responsiveness.  Automation  is  achieved  via  computer  program  or  software 
agents  to  perform  LLA  frequently,  and  in  near  real-time. 
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•  Discovery:  LLA  “discovers”  and  displays  a  network  of  word  pairs.  These  word 
pair  networks  are  characterized  by  one,  two,  or  three  word  themes.  The 
weight  of  each  theme  is  determined  based  on  its  frequency  of  occurrence.  It 
may  also  discover  blind  spots  of  human  analysis  that  are  caused  by  the 
overwhelming  data  for  human  analysts  to  consider. 

•  Validation:  LLA  may  provide  different  perspectives  of  links.  In  the  acquisition 
context,  links  discovered  by  human  analysts  may  emphasize  component  and 
part  connections  that  do  not  necessarily  reflect  content  overlaps.  LLA  looks 
for  the  overlapping  of  contents  to  help  identify  improved  affordability  and 
improved  response  to  meeting  warfighter  requirements,  and  achieve  better 
acquisition  decisions.  Consequently,  it  can  provide  improved  results  in  terms 
of  trust  and  quality  of  association  discovery  and  can  help  break  through  the 
taxonomy  of  ignorance  (Denby  &  Gammack,  1999)  and  organizational 
boundaries,  and  help  improve  organizational  reach. 

Research  Status 

Acquisition  Visibility  Portal  Background 

Our  goal  is  to  demonstrate  the  LLA  web  service  for  assisting  the  DoD-wide  effort  of 
integrating  and  maintaining  authoritative  and  accurate  acquisition  data  services  in  both 
legacy  and  new  platforms.  Specifically,  we  wanted  to  analyze  the  data  sources  from  the 
Acquisition  Visibility  Portal  (https://portal.acq.osd.mil)  by  examining  consistency,  correlation, 
and  gaps  among  categories  of  information  for  each  individual  program  listed  in  the  portal. 

One  of  the  biggest  risk  factors  in  defense  acquisition  is  the  unanticipated  effects  of 
program  interactions.  For  example,  ASD(SE)  and  Dahmann  worked  toward  identifying 
interdependence  among  programs  within  a  system  of  systems  (SoS).  Yet,  more  broadly, 
and  as  a  result  of  required  joint  capabilities,  portfolios  often  include  program 
interdependencies  and  system-of-systems  effects.  Ultimately,  the  current  “program-centric” 
acquisition  paradigm  is  increasingly  ill-suited  to  identify  and  address  program  risks  that  arise 
outside  of  program  boundaries.  LLA  can  help  isolate  these  issues  from  the  body  of 
information  collected,  which  have  yet  to  be  effectively  identified. 

To  begin  to  address  this  risk,  we  observed  that  very  little  of  the  information 
generated  for  program  oversight  is  amenable  to  effective  analysis.  Every  major  acquisition 
program’s  milestone  review  generates  volumes  of  information,  which  the  OSD  staff  is 
supposed  to  review  to  determine  if  the  program  is  properly  prepared  for  the  next  milestone. 
Although  they  are  beginning  to  compile  these  artifacts  centrally  to  facilitate  review  and 
analysis,  at  present,  the  only  way  to  analyze  the  information  in  these  artifacts  is  to  read 
them.  With  limitations  on  staffing,  little  time  is  available  to  thoroughly  review  these  artifacts. 
Moreover,  each  functional  community  is  required  to  review  only  the  particular  document  for 
which  it  is  responsible.  For  example,  the  systems  engineering  community  typically  only 
examines  the  Systems  Engineering  Plan  (SEP),  the  test  and  evaluation  community  looks 
only  at  the  Test  &  Evaluation  Master  Plan  (TEMP),  and  the  acquisition  community  looks  at 
the  Acquisition  Strategy  Report  (ASR).  Rarely  do  any  of  these  stakeholders  review  multiple 
reports  or  jointly  discuss  them  to  determine  if  they  are  mutually  consistent  and  to  consider 
inconsistencies  that  might  indicate  programmatic  risk.  There  is  even  less  incentive  and 
opportunity  to  look  for  external  factors  that  would  potentially  invalidate  the  assumptions  that 
underpin  the  basic  cost,  schedule,  and  performance  targets  of  each  program  execution. 

Overlaying  the  concept  maps  for  each  of  the  major  categories  of  artifacts  to  conduct 
a  pair-wise  comparison  might  expose  significant  disconnects  between  them.  We  are 
motivated  by  a  situation  in  which  the  SEP  identifies  a  critical  dependency  between  the 
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program  and  an  external  system,  but  the  TEMP  doesn’t  have  a  corresponding  reference  to 
testing  that  interdependency.  Therefore,  it  may  be  productive  to  compare  the  acquisition 
strategy  to  the  SEP  or  TEMP. 

Results 

LLA  maps  of  these  artifacts  from  one  category  to  another,  for  example,  the  SEP  at 
Milestone  B,  are  significantly  different  from  the  SEP  at  Milestone  C  that  might  indicate  a 
reduction  in  system  functionality  resulting  from  cost  increases  elsewhere.  These  maps, 
reported  as  themes,  concepts,  and  word  pairs,  may  help  cue  a  decision-maker’s  attention  to 
the  potential  issues  and  help  the  decision-maker  consider  specific  and  productive  directions 
for  further  scrutiny. 

To  develop  comprehensive  LLA  maps,  we  first  extracted  a  sample  from  a 
representative  MDAP  from  the  Acquisition  Visibility  Portal  (AVP)  with  categories  of 
information  to  demonstrate  the  method,  as  follows: 

•  SEP:  2  documents,  222  pages 

•  TEMP:  5  documents,  62  pages 

•  ASR:  1 1  documents  including  metrics,  634  pages 

•  SARs:  9  documents,  313  pages 

•  DAES:  19  documents,  447  pages 

•  Milestone  B  2366b  Certification  Acquisition  Decision  Memorandum  (ADM)  12 
documents,  105  pages 

•  APB:  3  documents,  39  pages 

•  TRA:  1  document,  1  page 

Figure  8  lists  the  top  20  themes  discovered  for  comparing  data  for  ASR  and  SEP 
with  the  highest  correlations.  In  Row  2,  there  are  299  word  pairs  for  the  two  sources 
together  classified  in  Theme  117(E);  47  of  them  appear  in  both  sources,  indicating  potential 
feature  overlaps.  The  correlation  is  the  ratio=47/299=0.157  which  indicates  15.7%  of  the 
features  represented  as  word  pairs  shared  in  both  artifacts.  As  a  detail  shown  in  Figure  9, 
part  of  299  word  pairs  in  Theme  117(E)  are  visualized  in  red,  yellow,  and  green  links, 
representing  the  shared  word  pairs,  unique  ones  to  ASR  and  SEP,  respectively. 
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73 
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55 
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65 
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508 

381 
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70 
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99 

56 
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14 
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62 

33 
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15 
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48 
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16 
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82 

49 
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17 

235(E) 
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329 
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42 
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18 

245(E) 
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39 

27 

0.096 

19 

113(E) 

334 
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57 

32 

0.096 

20 

310(P) 

586 

441 

90 

55 

0.094 

21 

182(A) 

197 

157 

22 

18 

0.091 

Figure  8.  Themes  for  Comparing  SEP  and  ASR,  Sorted  According  to  Correlation 

Ascending 

Figure  9  shows  that  there  are  concepts  related  to  these  word  nodes  that  appear 
uniquely  to  the  ASR  or  SEP. 

Since  the  SEP  document  is  supposed  to  support  the  ASR,  the  illustrations  and 
visualizations  of  it  might  inform  acquisition  professionals  about  why  concepts  in  the  SEP 
were  missing  from  the  ASR,  and  vice  versa. 
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Figure  9.  Detail  of  Word  Pairs  for  Theme  117(E):  Red  Links  for  Shared  Word  Pairs  for 
SEP  and  ASR  (Yellow  Links  for  Unique  Word  Pairs  Unique  to  ASR,  and  Green 
links  for  Unique  Word  Pairs  Unique  to  SEP) 

Figure  10  lists  the  least  correlated  themes  discovered  for  comparing  data  for  ASR 
and  SEP.  In  Row  2,  there  are  149  word  pairs  for  the  two  sources  together,  classified  in 
Theme  359(E)(A);  four  of  them  appear  in  both  sources  (overlap).  The  correlation  is  the 
ratio=4/149=0.027.  A  detail  shown  in  Figure  9,  part  of  149  word  pairs  in  Theme  359(A)  are 
visualized  in  red,  yellow,  and  green  links,  representing  the  shared  word  pairs,  unique  ones 
to  the  ASR  and  SEP,  respectively. 
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Figure  10.  Themes  for  Comparing  SEP  and  ASR,  Sorted  According  to  Descending 

Correlation 


Figure  1 1 .  Detail  of  Word  Pairs  for  Theme  359(A):  Red  Links  for  Shared  Word  Pairs  for 
SEP  and  ASR  (Yellow  Links  for  Unique  Word  Pairs  Unique  to  ASR,  and  Green 
Links  for  Unique  Word  Pairs  Unique  to  SEP) 
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In  Figure  11,  there  are  also  concepts  that  are  more  prevalent  in  the  ASR  than  in  the 
SEP.  The  ASR  includes  other  concepts  that  are  not  in  the  SEP  that  might  be  important. 

LLA  also  categorizes  themes  into  popular  (P),  emerging  (E),  and  anomalous  (A). 
Comparing  Figure  8  and  Figure  10,  one  can  see  that  popular  themes  tend  to  have  higher 
correlations  among  data  sources  (ASR  and  SEP),  while  anomalous  themes  tend  to  have 
lower  correlations  among  data  sources. 

For  each  pair  of  comparisons  for  two  categories  of  information,  we  use  the  ratio  of 
the  number  of  word  pairs  that  appear  in  both  categories  and  the  total  number  of  word  pairs 
as  an  overall  correlation  for  each  pair. 

In  Table  1,  the  highlighted  cells  are  the  ones  with  correlation  >  0.06.  The  categories 
DAES,  SARs,  and  SEP  have  higher  overall  correlations  with  other  ones.  The  most 
correlated  two  categories  are  SARs  and  DAES  (correlation  =  0.1 17).  The  category  TEMP 
has  the  lowest  overall  correlations  with  other  categories.  Although  TEMP  and  SEP  were 
both  produced  in  the  test  and  evaluation  community,  the  correlation  between  the  two  is  low 
(0.027). 


Table  1.  LLA  Correlations  Between  Categories  of  Information 


APB 

ASR 

2366B  Cert 

DAES 

SARs 

SEP 

TEMP 

TRA 

APB 

1.000 

0.007 

0.027 

0.022 

0.080 

0.014 

0.010 

0.005 

ASR 

0.007 

1.000 

0.015 

0.048 

0.025 

0.075 

0.028 

0.001 

2366B  Cert 

0.027 

0.015 

1.000 

0.026 

0.038 

0.026 

0.018 

0.068 

DAES 

0.022 

0.048 

0.026 

1.000 

0.117 

0.073 

0.023 

0.003 

SARs 

0.080 

0.025 

0.038 

0.117 

1.000 

0.044 

0.020 

0.004 

SEP 

0.014 

0.075 

0.026 

0.073 

0.044 

1.000 

0.027 

0.003 

TEMP 

0.010 

0.028 

0.018 

0.023 

0.020 

0.027 

1.000 

0.002 

TRA 

0.005 

0.001 

0.068 

0.003 

0.004 

0.003 

0.002 

1.000 

When  discussing  the  findings  with  the  domain  expert,  it  seems  the  correlation  is 
surprisingly  low  for  DAES  and  SARs.  DAES  and  SARs  reports  are  similar  in  context  and 
content  (both  relate  to  acquisition  performance);  they  would  be  expected  to  have  higher 
correlation.  Further  investigations,  such  as  the  following:  are  needed  to  see  what  might  be 
the  causes  for  the  low  correlation: 

•  To  investigate  if  significantly  different  content  appears  in  the  two  types  of 
reports;  for  example,  DAES  reports  may  include  more  details  than  SARs 
reports. 

•  To  differentiate  the  SAR  and  DAES  reports  by  year  and  compute  the 
correlations  over  time,  to  see  when  the  significant  discrepancies,  that  is,  the 
drop  in  the  correlation,  came  into  the  picture. 

•  To  correlate  the  DAES  or  SAR  reports  over  time  separately  to  see  if  the 
correlation  increases  and  decreases  might  have  to  do  with  the  new  features 
being  introduced  into  the  program,  and  therefore  correlate  to  the  significance 
of  low  or  high  changes  found  in  LLA  with  the  numeric  metrics  such  as  cost, 
schedule,  funding,  and  performance. 

Future  Work 

Since  this  is  the  first  program  to  have  undergone  a  relatively  comprehensive  LLA 
analysis  using  multiple  types  of  acquisition  documents,  the  findings  cannot  be  evaluated  in 
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terms  of  being  “good”  or  “bad,”  “normal”  or  “unusual,”  and  so  forth.  Therefore,  future 
investigation  should  consider  the  following  additional  studies: 

•  Analyze  additional  programs  in  the  AVP,  compute  the  correlation  matrices 
like  Table  1 ,  and  compare  the  results  to  determine  if  the  correlation  patterns 
are  similar  or  different. 

•  Discuss  the  findings  in  detail  with  the  domain  experts  and  personnel 
associated  with  the  programs  to  see  if  the  correlation  patterns  have 
significance,  as  follows: 

o  if  the  correlation  are  the  indications  for  data  quality  issues  and 

o  if  the  correlation  patterns  have  impacts  for  the  costs,  schedules, 
funding,  and  performance  of  the  programs. 

Conclusion 

In  this  paper,  we  demonstrated  how  to  apply  LLA  to  generate  maps  of  the  acquisition 
artifacts  among  multiple  categories  of  data.  These  maps,  reported  as  themes,  concepts,  and 
word  pairs,  may  help  identify  the  issues  and  offer  specific  and  productive  directions  for 
further  examination  as  to  why  there  are  gaps  among  the  categories  of  information. 
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Abstract 

This  research  attempted  to  capture  the  creative  aspects  of  government  program 
management  in  three  specific  areas:  efficiently  navigating  oversight,  capturing  the  intent  of 
regulations,  and  developing  innovative  risk  management  practices.  Respected  acquisition 
leaders  with  diverse  backgrounds  and  experiences  were  interviewed  with  ranks  ranging  from 
0-6  to  0-8  and  GS-15  to  SES.  Several  contractor  interviews  were  conducted  for  specific 
purposes.  The  data  were  iteratively  coded  and  analyzed  using  ATLAS. ti.  The  results  were 
categorized  into  four  themes,  each  with  three  sub-elements.  Differences  between 
respondents  with  program  director  experience  and  those  with  rapid  acquisition  experience 
are  discussed.  A  survey  was  then  distributed  to  the  interviewees  and  junior  acquisition 
professionals.  The  predominant  research  finding  is  that  senior  acquisition  professionals 
believe  that  relationship-building  is  of  paramount  importance.  This,  along  with  creative 
practices  regarding  how  to  externally  communicate  program  strategies,  greatly  increases  the 
probability  of  successfully  navigating  oversight  and  obtaining  waivers  or  tailoring  regulations. 
Various  risk  management  techniques  and  management  reserve  techniques  are  presented.  In 
addition,  knowledge  gaps  between  the  junior  acquisition  workforce  and  senior  leaders  were 
identified  based  on  statistical  significance  and  corrective  actions  recommended  where 
applicable.  Reports  and  outbriefs  were  developed,  tailored  to  each  class,  to  relay  these 
creative  practices  to  junior  acquisition  professionals. 

Introduction 

This  paper  presents  the  results  of  exploratory  thesis  research  regarding  creative 
program  management  practices  as  identified  by  senior  leaders.  For  the  purposes  of  this 
paper,  creative  is  defined  as  any  innovative,  resourceful,  uncommon,  or  out-of-the-box 
thinking  and  practices  leading  to  efficient  and  effective  program  management  without 
jeopardizing  integrity,  ethics,  or  laws.  The  literature  review  identified  three  areas  of 
investigation: 

Topic  1 :  How  to  creatively  reduce  non  value-added  oversight 
Topic  2:  How  to  creatively  capture  the  intent  of  regulations 
Topic  3:  Creative  practices  of  resource-loaded  risk  management 

The  first  two  topics  are  the  focus  of  this  paper  because  they  led  to  the  overarching 
findings.  Interviews  with  respected,  leading  practitioners  representing  diverse  programs  with 


1  This  study  is  an  original  product  developed  from  thesis  research  conducted  at  the  Air  Force  Institute 
of  Technology  (AFIT)  in  partial  fulfillment  of  a  Master  of  Science  in  Research  and  Development 
Management.  This  research  has  not  been  previously  published  and  is  not  under  consideration  by 
another  journal  for  publication. 
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varying  sizes  and  complexity  were  conducted.  A  survey  was  then  distributed  to  government 
acquisitions  personnel,  further  validating  interview  findings  with  quantitative  data,  as  well  as 
prioritizing  responses  from  senior  leaders  and  identifying  the  major  differences  in  the  junior 
workforce. 

Literature  Review 

Perhaps  the  greatest  impediment  to  the  achievement  of  high  quality — and 

productivity — is  . . .  burgeoning  bureaucracy. 

(Augustine,  1997,  p.  79) 

The  type  of  oversight  described  in  this  paper  must  be  defined  because  “oversight” 
can  have  various  meanings  based  on  the  reader’s  experiences.  For  the  purposes  of  this 
research,  oversight  consists  of  the  organizations  and  people  needed  to  approve  (either 
formally  or  informally)  a  program’s  approach  and/or  documentation  to  proceed  to  the  next 
phase  in  the  acquisition  life  cycle.  This  is  separate  from  government  oversight  of 
contractors  or  prime  contractor  oversight  of  subcontractors.  This  research  is  not  meant  to 
make  judgments  as  to  the  goodness  of  oversight  or  to  assess  the  theory  of  checks  and 
balances  versus  optimal  efficiency.  The  goal  is  to  identify  creative  ways  in  which  DoD 
acquisition  oversight  can  be  made  more  beneficial  or,  in  situations  when  oversight  is  overly 
cumbersome,  how  it  can  be  effectively  navigated  with  minimal  effort. 

Setting  the  Stage:  Extensive  Oversight — A  Serious  Issue 

Acquisition  oversight  began  in  the  1960s  (Acker,  1993).  Numerous  studies  and 
reports  on  defense  acquisition  have  subsequently  been  conducted  over  the  past  five 
decades.  A  common  theme  extracted  from  these  reports  is  that  a  serious  problem  exists 
with  extensive,  non-required,  and,  many  times,  non  value-added  oversight.  One  panel  of 
experts  estimated  the  cost  of  oversight  in  Air  Force  programs  to  be  as  high  as  $94  million 
(Neal,  2004).  Knue’s  (1991)  thesis  is  recommended  as  a  detailed  source  for  explaining 
oversight  of  and  within  the  DoD.  Additionally,  several  case  studies  exist  on  oversight  within 
Air  Force  programs.  A  few  of  the  more  prominent  reports  are  summarized  in  the  following 
paragraphs. 

Miller  and  Williams  (1993)  conducted  a  case  study  of  oversight  in  the  C-17  program. 
The  interviews  they  conducted  revealed  that  oversight  had  a  negative  effect  on  program 
management  and  morale.  There  was  “absolute  certainty  in  the  collective  consciousness”  of 
members  of  the  C-17  program  office  that  a  link  exists  between  oversight  and  its  effect  on 
cost  and  schedule  performance  (Knue,  1991,  p.  72).  Interviewees  also  cited  external 
(outside  the  chain  of  command)  sources  of  oversight  from  nine  distinct  organizations  that 
negatively  affected  the  program.  These  nine  external  sources  did  not  include  legislative, 
executive,  and  media  oversight  (Miller  &  Williams,  1993). 

A  RAND  study  of  the  B-1B  bomber  program  concluded  that  an  extraordinary  amount 
of  internal  and  external  coordination  was  required,  leading  to  a  “ceaseless  series  of 
meetings,  calls,  and  memos”  (Bodilly,  1993,  p.  40).  The  study  concluded  that  14  different 
groups  had  major  roles  in  the  program. 

The  Beyond  Goldwater-Nichols  Phase  2  Report  (Murdock  et  al.,  2005)  stated  that 
the  “well-intentioned  majority  of  the  acquisition  corps  today  faces  two  significant  types  of 
bureaucratic  impediments:  highly  centralized  oversight  and  conflicting  guidance”  (p.  91). 

The  Phase  2  Report  also  found  that  program  managers  (PMs)  and  program  executive 
officers  (PEOs)  are  left  with  about  50%  or  less  of  their  time  to  actually  manage  their 
programs  (Murdock  et  al.,  2005). 
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The  highly  regarded  Defense  Acquisition  Performance  Assessment  (DAPA)  report  in 
2006  showed  that  97%  of  the  survey  inputs  received  indicated  that  the  current  oversight  and 
leadership  process  is  deficient  (Kadish  et  al. ,  2006).  Figure  1 ,  from  the  DAPA  report, 
highlights  the  key  issues  affecting  government  acquisitions.  As  can  be  seen  in  the  figure, 
respondents  viewed  oversight  as  the  most  prevalent  issue. 


INTEGRATED  LOOK  AT  KEY  ISSUES 


Figure  1.  Integrated  Look  at  Key  Issues 

(Kadish  et  al.,  2006) 

Oversight  is  discussed  in  several  sections  of  the  DAPA  report.  Figure  2  is  a  one- 
page  summary  of  the  myriad  DAPA  findings  with  respect  to  oversight.  Issues  relating  to 
oversight  are  divided  into  four  categories:  Extent  of  Oversight,  Programmatic  Issues, 
Accountability/Authority  Issues,  and  Effect  on  Progress. 
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Extent  of  oversight 

-  Current  oversight  process  is  burdensome,  ineffective,  adds  little  value,  and  inhibits 
steady  improvement 

-  Excessive  numbers  of  reviews  and  oversight  personnel;  quantity  replaced  quality 

-  Regulations  written  to  implement  policy  are  more  stringent  than  the  policy  itself 

-  Dissatisfaction  with  sheer  volume  of  acquisition  laws,  regulations,  and  policies 

-  Rely  on  overlapping  layers  of  reviews  at  the  expense  of  focus  and  quality 

Programmatic  Issues 

-  Acquisition  Category  (ACAT)  designation  process  results  in  excessive  number  of 
programs  requiring  additional  level  of  DAB  approvals,  causing  excessive  reporting 
requirements 

-  Even  with  the  laborious  and  extensive  oversight,  troubled  programs  still  pass  through 

-  Lack  of  continuity  or  attendance  on  OSD  acquisition  IPTs  results  in  the  re-emergence  of 
issues  previously  resolved  and  revisiting  decisions 

-  Policy  and  guidance  often  conflict,  resulting  in  ignoring  policy  or  seeking  legal  advice 

-  Institutional  biases  toward  waiving  or  tailoring  regulations  (even  though  DoD  Directives 
promote  tailoring  for  each  program's  situation) 

Accountability/Authority  Issues 

-  Oversight  is  preferred  to  accountability  and  based  on  a  lack  of  trust 

-  Oversight  dilutes  or  eliminates  accountability  for  program  performance 

-  PMs effectiveness  is  constrained  by  people  who  do  not  share  responsibility  or 
accountability 

-  OSD  staff  do  not  have  decision-making  authority  or  timely  access  to  principal  decision 
makers 

-  None  of  the  review  bodies  are  accountable  for  the  impact  of  the  changes  they  imposed 

Progress  Suffers 

-  Staffs  allowed  to  assume  de-facto  program  authority,  stop  progress  and  increase 
program  scope 

-  Programs  advance  in  spite  of  the  oversight  process  rather  than  because  of  it 

-  PM  does  not  have  authority  to  bypass  a  stakeholders  "no"  vote,  programs  progress  held 
hostage 


Figure  2.  Summary  of  DAPA  Report  Findings  on  Oversight 

(extracted  from  Kadish  et  al.,  2006) 

Lastly,  Ford,  Colburn,  and  Morris  (2012)  found  that  large  programs  and  budgets, 
such  as  acquisition  category  (ACAT)  1  multi-year  programs,  are  easy  targets  for  increased 
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oversight  and  longer  approval  chains.  They  showed  a  positive  correlation  between  program 
size  (measured  by  budget  dollars)  and  the  extent  of  oversight. 


Factors  Affecting  the  Level  of  Oversight 

A  factor  affecting  one’s  ability  to  manage  oversight  and  stakeholders  is  political  skill. 
Political  skills  include  developing  coalitions  and  gaining  resources,  assistance,  and 
approvals  from  senior  leaders  and  other  relevant  parties  (Yukl,  2006).  Additionally,  De  Wit 
(1988,  p.  167)  stated,  “political  skill  will  be  a  useful  attribute  on  the  part  of  the  project 
manager  to  assure  maximum  satisfaction  among  the  stakeholders.  This  is  of  special 
importance  on  public-sector  projects.”  Furthermore,  Yukl  (2006)  discusses  five  skills 
required  for  leading  cross-functional  teams  (which  includes  integrated  product  teams  [I PT sj). 
Specifically,  political  and  interpersonal  skills  are  associated  with  managing  oversight  and 
leading  IPTs  (Yukl,  2006).  These  skills  involve  understanding  the  needs  and  values  of 
stakeholders  to  influence  them  and  resolve  conflict.  In  addition,  a  higher  program 
classification  can  reduce  oversight  because  it  limits  the  number  of  people  to  those  with  the 
requisite  security  classification  and  need  to  know  (Ford,  Colburn,  &  Morris,  2012). 


Finally,  the  literature  on  DoD  acquisitions  points  to  four  main  areas  that  affect 
oversight  (Pagliano  &  O’Rourke,  2004;  Kadish  et  al,  2006).  The  first  factor  affecting 
oversight  is  uncertainty.  If  all  else  is  constant,  the  greater  the  program  uncertainty,  the  more 
extensive  the  oversight  will  be.  Second,  oversight  will  increase  as  program  criticality 
increases.  In  other  words,  if  a  program  is  critical  to  national  security,  a  high  degree  of 
oversight  will  exist.  Third,  oversight  will  increase  as  trust  decreases.  If  the  chain  of 
command  and  external  stakeholders  do  not  have  a  high  degree  of  trust  in  what  the  program 
office  is  doing,  more  external  reviews  and  proof  will  be  required  from  the  program  office, 
thus  leading  to  more  extensive  oversight.  Finally,  oversight  will  increase  as  the  level  of 
control  and  standardization  from  leaders  increases.  A  model  was  developed  (Figure  3)  from 
the  review  showing  how  various  factors  affect  the  level  of  oversight  in  a  program. 


Instability  of  bu  dgetv  fun  ding 
Evolutionary  Acquisition  concept 
(incremental  capability  delivenes) 
Novelty  of  program 
Instability  of  cost  predictions 
Level  of  uncertain ty 
Level  of  program  criticality 
Level  of  control  and  standardization 


— >  Level  of  oversight 


Political  skills 
Level  of  trust 

Program  classification  level 


(-) 


>  Level  of  oversight 


*£ SL 

(+)=  positvelyco  related  to 
B  =  negatrvelycorrelatedte 


Figure  3.  Factors  Affecting  Level  of  Oversight 
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Methodology 
Research  Design 

Theoretical  Method 

This  research  utilized  Grounded  Theory  Methods  (GTM).  Auerbach  and  Silverstein 
(2003)  suggest  GTM  when  a  researcher’s  particular  theory  is  at  its  early  stage,  not  enough 
is  known  to  state  hypotheses  prior  to  the  investigation,  and  the  major  research  involves 
identifying  and  categorizing  elements  to  explore  their  connections.  One  of  the  key  tenets  of 
GTM  is  the  iterative  process  of  collecting,  coding,  and  interpreting  the  data,  also  known  as 
analytic  induction  (Binder  &  Edwards,  2010).  As  such,  the  interview  process  and  data 
analysis  were  iterative  in  nature. 

Sample  Size 

For  the  interview  sample  size,  Eisenhardt  (1989)  states  that  4-10  cases  have 
worked  well  for  most  qualitative  studies.  Separate  research  conducted  by  Ellram  (1996) 
identifies  6-10  cases  as  sufficiently  large  for  evaluation  and  empirical  grounding.  Therefore, 
one-on-one  interviews  were  conducted  with  10  hand-picked  senior  acquisition  leaders  with 
diverse  backgrounds  and  program  experience. 

Sampling  Strategy 

Eisenhardt  (1989,  p.  537)  states  that  “random  [case]  selection  is  neither  necessary, 
nor  preferable”  when  building  theory  from  case  studies.  Both  purposive  and  snowball 
sampling  were  used  in  this  research.  Purposive  sampling  is  used  in  qualitative  research 
where  individuals  are  selected  based  on  their  ability  to  better  inform  the  researcher 
(Krathwohl,  1998;  Patten,  2009).  Snowball  sampling  entails  identifying  future  participants 
based  on  recommendations  from  past  participants  (Krathwohl,  1998).  In  other  words,  the 
interviewees  specifically  suggest  other  people  to  interview.  Snowball  sampling  successfully 
led  to  three  interviews. 

Personal  Interviews 

The  population  for  this  research  consisted  of  Air  Force  program  managers  (PMs) 
with  at  least  20  years  of  experience.  This  included  active  duty  and  retired  officers  with  ranks 
ranging  from  colonel  to  major  general,  active  duty  civilians  with  ranks  ranging  from  GS-15  to 
Senior  Executive  Service  (SES),  and  three  government  contractors.  Both  Air  Force  product 
centers,  the  Life  Cycle  Management  Center  (AFLCMC)  and  Space  and  Missile  Systems 
Center  (SMC),  were  represented,  along  with  Special  Operations  Command  (SOCOM). 
Programs  covered  included  Global  Positioning  System  (GPS),  SOCOM  Fixed  Wing, 
Spacelift  Range,  Big  Safari,  F-22,  Project  Dragon  Spear,  Military  Satellite  Communications 
Directorate  (MILSATCOM),  FalconSAT,  and  the  Secretary  of  the  Air  Force  for  Acquisition 
(SAF/AQ)  and  Aerospace  organizations. 

Coding:  Atlas. ti 

The  ExpressScribe  program  was  used  to  quickly  transfer  the  interviews  into 
Microsoft  Word  documents.  The  interviews  were  then  coded,  categorized,  and  analyzed  in 
ATLAS.ti,  a  software  program  specifically  designed  for  qualitative  research,  using  an  “open 
coding”  of  labels  to  extract  major  themes.  All  responses  were  analyzed  for  common 
themes.  Three  rounds  of  analysis  were  conducted  in  ATLAS.ti. 

Survey 

Additionally,  a  survey  was  developed  from  the  interview  data  and  distributed  to  the 
interviewees  as  well  as  junior  officers  and  civilians  in  the  introductory  Fundamentals  of 
Acquisition  Management  (FAM)  103  and  mid-level  Intermediate  Program  Management 
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(IPM)  301  skills  courses.  The  survey  contained  65  questions  on  a  1-5  Likert  scale  with  an 
additional  column  for  respondents  to  mark  “unknown.”  Two  classes  from  each  course  were 
surveyed.  Fifty-eight  students  in  the  FAM  103  courses  and  35  students  in  the  IPM  301 
courses  provided  usable  surveys,  totaling  93.  The  survey  served  three  purposes: 

1 .  Quantitatively  validate  interview  responses  with  statistical  significance 

2.  Prioritize  themes  from  senior  leaders 

3.  Identify  knowledge  gaps  in  the  junior  workforce 

According  to  Cohen  (1992),  for  an  alpha  (a)  level  of  0.05  (a  95%  confidence  level) 
and  a  medium  effect  size,  one  must  have  a  sample  size  of  at  least  85.  For  a  large  effect 
size  at  the  same  confidence  level,  the  sample  size  should  be  at  least  28.  Therefore,  a 
conservative  sample  size  of  at  least  85  was  the  goal;  93  usable  student  surveys  were 
completed  along  with  the  additional  10  from  the  senior  leaders. 

Limitations/Assumptions 

The  nature  of  qualitative  data  and  grounded  theory  research  allows  for  interpretation 
depending  on  the  researcher’s  point  of  view.  Qualitative  analysis  “can  therefore  become 
biased  based  on  individual  experience  and  perspective”  (Ford  et  al.,  2012).  The  author 
endeavored  to  be  cognizant  of  bias  and  avoid  it  when  guiding  interview  discussions  and 
interpreting,  coding,  and  analyzing  the  data. 

The  results  will  have  a  high  degree  of  reliability  for  all  DoD  program  managers,  even 
though  the  population  set  was  limited  to  Air  Force  program  managers.  Studies  have  shown 
that  all  the  Services  are  comparable  with  respect  to  their  acquisition  processes  and  record 
of  success  (Kadish  etal.,  2006;  Burton,  1993). 

Analysis  and  Results 

Interview  Analysis 

From  three  iterative  rounds  of  coding  the  data,  four  themes  and  12  sub-elements 
emerged  as  shown  in  Figure  4. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  175- 


Creative  Program  Management  Results 

Theme  1:  Relationship-building: 

1 .  Break  down  barriers  and  build  relationships 

2.  External  communications  strategy 

3.  Contractor  relationships 

Theme2:  Creative  Practices: 

1.  Efficiencies  Time  Savers 

2.  Seeking  waivers  tailoring  regulations 

3.  Navigating  Oversight 

Theme  3:  Current  Acquisition  Environment: 

1 .  Culture  of  Compliance 

2.  Current  acquisition  professional  roles  and  focus  areas 

3.  Acquisition  doctrine  and  lessons  learned 

Theme  4:  Risk  Management  Principles: 

1 .  Enterprise  Risks 

2.  Working  risks 

3.  Management  Reserve  Principles 


Figure  4.  ATLAS.ti  Round  3  Results 

A  co-occurrence  table  was  developed  analyzing  where  common  occurrences  within 
and  between  themes  and  codes  occurred.  The  strength  of  a  co-occurrence  is  affected  by 
the  number  of  times  a  comment  was  made  either  during  a  single  interview  or  between 
several  interviews.  Strong  and  medium  co-occurrences  are  collected  and  displayed  in  Table 
1 ,  with  the  three  key  findings  for  this  paper  highlighted. 


Table  1.  Strong  and  Medium  Co-Occurrences  Between  Sub-Elements 


Strong  Co-occurrences 

Break  down  barriers  &  build  relationships 

strongly  co-occurs  with 

External  communications  strategy 

Break  down  barriers  &  build  relationships 

strongly  co-occurs  with 

Navigating  oversight 

Medium  Co-occurrences 

Break  down  barriers  &  build  relationships 

co-occurs  with 

Contractor  relationships 

Break  down  barriers  &  build  relationships 

co-occurs  with 

Seeking  waivers/tailoring  regulations 

External  communications  strategy 

co-occurs  with 

Efficiencies/time  savers 

External  communications  strategy 

co-occurs  with 

Seeking  waivers/tailoring  regulations 

External  communications  strategy 

co-occurs  with 

Navigating  oversight 

Efficiencies/time  savers 

co-occurs  with 

Navigating  oversight 

Enterprise  risks 

co-occurs  with 

Working  risks 

Working  risks 

co-occurs  with 

Management  reserve  principles 

The  interviews  were  also  categorized  based  on  the  respondents  with  experience  as 
a  program  director  (PD)  and  those  with  experience  in  rapid  acquisitions.  Five  interviews 
were  coded  as  those  with  PD  experience  and  three  interviews  were  coded  as  those  with 
rapid  acquisition  experience.  Figure  5  graphically  displays  the  focus  areas  between  the  two 
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groups.  Interestingly,  the  top  three  responses  were  the  same  for  both  groups.  These  were 
the  External  Communications  Strategy,  Break  Down  Barriers  and  Build  Relationships,  and 
Navigating  Oversight.  The  main  focus  area  for  the  program  directors  regarded  their  external 
communications  strategies,  which  is  understandable  given  the  amount  of  oversight  and 
number  of  stakeholders  present  in  MDAP  programs.  A  great  deal  of  time  is  spent  ensuring 
goals  and  strategies  are  being  communicated  clearly,  accurately,  and  in  a  timely  manner 
across  organizational  boundaries.  Navigating  Oversight  was  the  second  focus  area  for  both 
program  directors  and  those  with  rapid  acquisition  experience.  However,  a  key  difference 
exists  between  the  two  groups.  Program  directors’  practices  relating  to  oversight  involved 
how  to  efficiently  and  effectively  work  through  the  current  oversight  and  regulations.  The 
oversight  was  viewed  more  as  a  fact  of  life  that  had  to  be  worked  through.  In  contrast,  rapid 
acquisition  responses  focused  more  on  how  to  circumvent  the  oversight  from  the  start.  In 
other  words,  rather  than  trying  to  efficiently  work  through  oversight,  rapid  acquisition 
organizations  delegate  approvals  and  obtain  waivers  from  the  beginning  (the  thesis  contains 
a  case  study  on  how  USSOCOM  instantly  tailors  5000.02  via  SOCOM  Directive  70-1 ). 
Accepting  the  oversight  level  and  figuring  out  how  best  to  navigate  it  is  very  different  than 
navigating  oversight  by  avoiding  the  oversight  from  the  beginning. 


Histogram  of  focus  areas  between 
PDs  vs  Rapid  Experience 


Figure  5.  Histogram  Comparing  PD  and  Rapid  Experience  Responses 

Additionally,  a  significant  difference  also  existed  between  the  PD  and  rapid 
experience  responses  for  Seeking  Waivers/Tailoring  Regulations.  Rapid  acquisition 
organizations  spend  a  lot  of  effort  on  tailoring  programs  and  obtaining  waivers.  However, 
program  directors  often  viewed  the  process  of  obtaining  a  waiver  as  more  difficult  than 
actually  complying  with  the  guidance,  even  if  it  did  not  make  sense  for  the  program. 
Therefore,  program  tailoring  was  a  larger  focus  area  for  those  with  rapid  acquisition 
experience.  Figure  6  provides  a  decision-making  process  to  obtain  a  waiver/tailoring  based 
on  the  interviews  in  the  “Seeking  Waivers/Tailoring  Regulations”  sub-element. 
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Figure  6.  Decision-Making  Process  to  Obtain  a  Waiver/Tailoring 
Survey  Analysis 

Table  2  shows  the  overall  survey  data  results  divided  into  junior-level  and  senior- 
level  responses.  Of  particular  note  for  the  results  discussion  is  the  percentage  of  “unknown” 
responses  from  students  in  each  sub-element,  some  of  which  were  unexpected. 


Table  2.  Overall  Survey  Results 


Junior  Responses 

Mean 

St.  Dev 

95%  C.I.** 

%  Unknown 

Break  Down  Barriers  and  Build  Relationships 

3.746 

0.343 

3.06-4.43 

7.26% 

External  Communl cations  Strategy 

4.088 

0.343 

3.40-4.77 

1.88% 

Navigating  Ove  rsl  ght 

3.551 

0.463 

262-4.48 

19.47% 

Sen 

ior  Resoons 

es 

Mean 

St.  Dev 

95%  C.I.** 

%  Unknown 

Break  Down  Barriers  and  Build  Relationships 

4.397 

0.361 

3.68-5 

N/A 

External  Communications  Strategy 

4.500 

0.082 

4.34-4.66 

N/A 

Navigating  Ove  rsl  ght 

4.056 

0.513 

3.03-5 

N/A 

*  Threw  out  Question  #3  and  #63  --  poorly  worded/mis-leading 

**  Juniorresponsesin9S%C.I.  (Confide nee  Interval) range  due  to  Empirical  Rule  (93  respondents) 

**  Senior  responses  in  95%C.I.  range  due  to  passing  Shapiro-Wilk  test  of  normality 

An  Analysis  of  Variance  (ANOVA)  was  conducted  for  each  sub-element.  Both  Break 
Down  Barriers  and  Build  Relationships  and  Navigating  Oversight  showed  ANOVA 
significance  at  the  98%  confidence  level.  Normality  is  required  from  both  groups  for  a  valid 
ANOVA  test.  Normality  can  be  assumed  for  the  students’  responses  because  a  random 
sample  of  93  data  points  was  collected  and  used  (normality  requires  at  least  30  data  points 
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collected  at  random  from  the  population;  McClave,  Benson,  &  Sincich,  2010).  However, 
because  only  10  data  points  were  used  for  the  senior  leaders  group,  a  Shapiro-Wilk  test  of 
normality  was  conducted  on  the  three  sub-elements  with  significant  results.  Navigating 
Oversight  showed  normality  by  having  a  Shapiro-Wilk  value  greater  than  0.05.  Initially, 
normality  was  not  shown  for  the  Break  Down  Barriers  and  Build  Relationships  sub-element, 
but  after  investigation  one  survey  response  was  removed  with  high  confidence  that  the 
respondent  accidentally  reverse  coded  one  of  the  questions  (answered  1  instead  of  5  on  the 
Likert  scale)  based  on  their  interview  remarks.  After  this  was  done,  this  sub-element  passed 
the  Shapiro-Wilk  test,  showing  normality  as  well. 

Overview  of  Theme  and  Sub-Element  Results 

The  three  key  sub-elements  were  pulled  from  the  results  and  are  presented  next. 
Figures  7-9  give  an  overall  assessment  for  each  sub-element.  The  overall  assessment 
consists  of  two  parts.  A  qualitative  assessment  rating  of  1  to  5  is  given  based  on  the 
interviews  and  ATLAS.ti  analysis  (consistency  and  quantity  of  quotes,  importance  placed  on 
quotes,  number  of  co-occurrences,  strength  of  co-occurrences,  and  other  subjective 
measures).  Additionally,  quantitative  top-level  survey  results  are  provided.  The  average 
response  is  on  a  1  to  5  Likert  scale  from  the  survey,  and  the  percent  unknown  is  the  percent 
of  respondents  that  marked  unknown  for  questions  relating  to  each  particular  sub-element. 
Lastly,  a  “Yes”  or  “No”  is  given  if  the  ANOVA  test  between  the  Junior  and  Senior  responses 
for  that  sub-element  was  significant. 

Theme  1  Sub-Element  1:  Break  Down  Barriers  and  Build  Relationships 


Qualitative  Assessment:  5 

Survey  Results: 

Junior: 

Senior: 

Avg  response: 

3.75 

4.40 

%  Unknown: 

7.3% 

N/A 

ANOVA  Significant? 

Yes 

Figure  7.  Overall  Assessment  for  Theme  1  Sub-Element  1 

•  Building  personal,  trusting  relationships  requires  consistency  and  stability 

•  Importance  of  following  through  on  your  word 

•  Importance  of  networking  plus  solid  rationale 

•  Returning  un-executable  money  builds  trust  in  large  programs 

Building  relationships  and  trust  was  the  most  commonly  vocalized  point  throughout 
the  interviews  when  discussing  how  best  to  navigate  oversight  or  obtain  a  waiver  or 
tailoring.  Building  and  maintaining  strong,  trusting  relationships  with  peers,  co-workers, 
superiors,  stakeholders,  and  various  members  of  oversight  is  a  continual  process  built  over 
time.  Trust  is  increased  when  project  members  follow  through  on  their  word.  Although 
intuitive,  the  importance  of  doing  what  you  say  you  will  do,  when  you  said  you  would  do  it, 
should  not  be  undervalued. 

Personal  relationships  with  a  high  degree  of  trust  require  consistency  and  stability, 
which  is  often  lacking  in  major  acquisition  programs.  Air  Force  military  PM  tenure  is  typically 
a  three-year  tour  for  the  actual  materiel  leader  billet.  Below  the  PM  level,  military  acquisition 
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officers  and  engineers  are  usually  in  a  program  for  two  years  and  then  do  a  permanent 
change  of  assignment  (PCA)  in  which  they  switch  jobs,  which  can  be  within  the  same 
program  office  or  not.  Even  if  military  members  prefer  to  stay  in  their  assignments,  it  may 
not  be  good  for  their  career  to  do  so.  The  two  years  does  not  include  any  training, 
continuous  learning,  deployments,  or  additional  duties  the  member  might  need  to  complete. 
One  PM  the  author  previously  worked  with  stated  the  turnover  issue  clearly.  Simply  put, 
they  lost  half  their  people  every  summer,  and  that  was  a  best  case  scenario.  Worst  case, 
they  had  a  complete  turnover  one  year  in  which  no  military  continuity  existed  in  a  major 
ACAT  I  program.  Stability  and  consistency,  and  the  resultant  trust  and  relationships,  are 
constricted  by  the  acquisition  assignments  process.  Alternatively,  organizations  with  a  rich 
history  and  culture,  such  as  Big  Safari,  with  only  three  or  four  directors  in  the  past  60  years, 
allow  for  close,  personal  relationships  to  be  cultivated  over  time. 

Networking  is  extremely  vital  to  get  one’s  issue  “brought  to  the  table.”  As  one 
respondent  mentioned,  “I  would  have  never  been  promoted  once  in  my  life  if  it  wasn’t  about 
relationships  ....  I  built  relationships,  I  knew  what  people  wanted,  I  knew  the  people  to  rely 
on,  I  did  the  extra  thing,  so  relationship-building  in  that  oversight  process  is  instrumental.” 
Networking  builds  trust  by  building  closer  relationships.  This  in  turn  increases  the  likelihood 
for  a  program  approval,  waiver,  or  tailoring.  However,  some  negative  aspects  of  networking 
were  cited  in  the  interviews  as  well.  When  one  becomes  more  senior  and  is  on  their  second 
or  third  tour  at  the  same  base,  the  people  who  have  previously  known  them  may  still  view 
them  as  their  company  grade  officer  (CGO)  friend  and  not  show  the  requisite  respect. 
Additionally,  past  co-workers  may  not  be  as  concerned  about  deadlines  because  they  have 
a  personal  relationship  with  the  senior.  Last,  the  ease  of  recognizing  “phony  networking” 
was  cited  in  a  couple  interviews,  which  is  when  one  realizes  someone  is  building  a 
relationship  solely  for  their  own  benefit.  Although  drawbacks  to  networking  exist,  the 
positive  aspects  far  outweigh  the  drawbacks. 

Building  relationships  is  enabled  by  knowing  what  you  are  doing.  Even  if  all  the 
previous  statements  were  true,  if  the  rationale  for  what  you  are  trying  to  do  is  flimsy,  trust 
and  networking  will  be  far  less  effective.  Having  solid  rationale  in  your  decision-making  is  a 
key  enabler  to  building  trust  because  others  may  not  want  to  enable  members  of  their  own 
network  to  assist  in  doing  something  that  does  not  make  sense  if  it  will  result  in  a  lower  trust 
level  for  them.  As  one  respondent  discussed,  “Having  a  sense  of  purpose,  knowing  what 
you’re  trying  to  do,  and  having  strong  rationale  communicates  a  message  much  better.” 

Lastly,  returning  un-executable  money  builds  trust  in  large  programs,  if  they  are 
behind  schedule  and  must  do  so.  The  money  must  be  returned  through  the  PEO,  not 
directly  to  Air  Force  or  other  channels.  Returning  un-executable  money  does  not  include 
“expired”  funds. 

In  this  sub-element,  the  responses  between  the  students  and  senior  leaders  were 
significantly  different.  The  mean  of  the  senior  responses  was  4.40  compared  to  a  mean  of 
3.75  for  students.  As  was  briefed  to  each  FAM  and  I  PM  class,  the  senior  leaders 
emphasized  and  put  much  more  value  on  relationship-building,  building  trust,  and 
networking  than  did  the  students.  The  takeaway  for  the  students  is  that  as  they  are  starting 
out  or  continuing  their  careers,  they  should  begin  building  relationships  with  folks  in  required 
trainings,  other  programs,  outside  of  work,  etc.,  to  expand  their  network.  Of  course,  this 
cannot  be  done  from  a  selfish  or  “further  myself”  point  of  view,  but  rather  should  be 
genuinely  for  the  benefit  of  all. 

In  summary,  as  one  respondent  discussed,  “What  do  I  do  to  navigate  [oversight]?  I 
try  to  break  down  those  barriers  as  much  as  possible.  I  really  try  to  build  relationships  with 
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people,  so  that  they  know  if  something  is  really  bugging  them  they  can  give  me  a  call  so  we 
can  talk  back  and  forth.” 

Theme  1  Sub-Element  2:  External  Communications  Strategy 


Qualitative  Assessment:  5 

Survey  Results: 

Junior: 

Senior: 

Avg  response: 

4.09 

4.5 

%  Unknown: 

1 .9% 

N/A 

ANOVA  Significant? 

No 

Figure  8.  Overall  Assessment  for  Theme  1  Sub-Element  2 

•  “Walking  the  building”  every  time 

•  Benefits  of  physical  communications 

•  “Ground  swell”  or  “burning  your  boots” 

•  Value  of  an  elevator  speech 

•  Knowing  and  communicating  the  “views  of  others” 

•  Ability  to  communicate  across  paradigms 

Once  a  decision  is  made  as  to  the  strategy  on  an  issue,  how  the  PM  externally 
communicates  and  “sells”  what  they’re  doing  is  very  important.  Several  interviewees 
provided  approaches  they  take.  These  include  “walking  the  building”  each  time  the  PM  is  at 
the  Pentagon,  physical  communications,  and  “ground  swell”  or  “burning  your  boots” 
(proactive  staff  communication  and  dissemination  of  program  strategies).  Also,  the  value  of 
an  elevator  speech,  knowing  the  “views  of  others,”  and  the  ability  to  communicate  across 
paradigms  all  go  a  long  way  toward  effectively  communicating  what  the  program  is  trying  to 
accomplish.  Additionally,  this  sub-element  had  over  a  2:1  ratio  of  responses  from  program 
directors  versus  respondents  with  rapid  experience.  In  general,  those  with  PD  experience 
put  much  more  emphasis  into  the  importance  and  value  of  communicating  what  they  are 
doing.  The  likely  reason  for  this  is  because  large  ACAT  I  programs  experience  much  more 
oversight  (due  to  the  multi-year,  high-dollar  value,  and  industry  and  congressional 
stakeholders)  than  smaller,  more  rapid  programs.  However,  in  ACAT  I  programs,  decision¬ 
making  and  oversight  require  more  stakeholder  analysis  and  often  consist  of  a  “one-shot” 
opportunity  to  obtain  program  approvals,  thus  leading  to  the  higher  importance  of  the 
program’s  external  communications  strategy  from  program  directors. 

Several  respondents  mentioned  how  they  “walk  the  building”  when  they  are  visiting 
Washington,  DC.  This  term  is  used  to  describe  how  a  PM  should  visit  key  stakeholders, 
members  of  oversight,  and  members  of  their  network  when  walking  around  the  Pentagon. 

In  particular,  they  should  do  this  each  time  they  are  there,  especially  when  nothing  is 
needed  from  the  people  they  are  visiting.  Visiting  offices  and  asking  folks  if  they  need 
anything  from  you  helps  build  trust  and,  with  noble  intent  all  along,  can  enable  reciprocal 
generosity  when  you  need  something  from  them.  In  other  words,  a  genuine,  proactive  offer 
to  help  others  without  any  expectation  for  them  to  reciprocate  in  the  future  is  an  effective 
communication  strategy  to  build  long-term  relationships. 
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Additionally,  physical  communications  are  far  better  than  electronic  means.  Physical 
communications  enable  one  to  match  a  face  with  a  name,  increase  the  importance  of  the 
issue  (if  one  flies  to  discuss  an  issue  rather  than  e-mailing  or  calling,  they  are  putting  higher 
importance  on  the  issue),  and  make  it  more  difficult  to  ignore  the  issue.  Ignoring  an  e-mail 
is  fairly  easy  and  ignoring  a  phone  call  is  not  much  harder.  However,  when  someone 
physically  visits  you  to  discuss  an  issue,  and  then  comes  back  to  discuss  the  results,  it  is  far 
more  difficult  to  ignore  that  person’s  requests. 

Another  way  to  externally  communicate  a  strategy  is  by  “ground  swell”  or  “burning 
your  boots.”  This  refers  to  the  program  staff,  predominantly  the  Program  Element  Monitor 
(PEM),  proactively  communicating  and  disseminating  the  strategy  and  goals  throughout  the 
myriad  program  stakeholders  in  Washington,  DC.  This  is  done  by  working  the  staffing  and 
issues  from  the  ground  up,  communicating  to  all  stakeholders  and  staffs  first  so  that  there 
are  no  surprises  and  so  that  any  possible  issues  are  brought  to  light  early  on.  As  one 
respondent  mentioned,  “really  good  action  officer  work  can  save  hours  upon  hours  of 
wasted  time  in  meetings.” 

Business,  organizational  behavior,  and  management  books  often  discuss  the 
importance  of  an  elevator  speech  (albeit  using  different  terms).  The  premise  is  that  if  you 
were  to  find  yourself  riding  in  an  elevator  with  a  senior  manager,  you  should  always  have  a 
short  (~1-2  minute)  speech  or  talking  points  in  mind  to  gain  the  senior  manager’s  support  in 
the  time  it  takes  to  ride  in  the  elevator.  Interviewees  discussed  the  importance  of  this 
concept  in  acquisitions  as  well,  with  some  discussing  the  value  of  a  hard-hitting  one-liner. 
PMs  need  to  have  a  short,  direct,  and  effective  means  to  communicate  the  program 
capability  and  its  vital  importance  without  going  into  highly  technical  or  programmatic  details. 
As  one  respondent  said,  “When  I  was  having  a  problem  getting  funding  for  xx  program,  I 
met  with  a  key  staffer.  I  said  to  him  ‘Do  you  want  our  enemies  to  be  able  to  launch  a  nuke 
at  us  and  we’re  not  able  to  detect  it  early  enough  to  destroy  it?”’  ‘Well,  no.’  This  program 
ensures  early  warning  to  protect  the  homeland.  Period.’”  These  statements  should  be  clear 
and  concise  to  the  maximum  extent  possible.  An  excellent  one-liner  can  be  crucial  for  three 
reasons: 


1 .  if  one  unexpectedly  has  a  moment  of  the  senior’s  time; 

2.  to  translate  a  technical  program  into  a  tangible,  national  security  issue;  and 

3.  in  helping  the  oversight  help  the  program. 

Staff  Summary  Sheets  (SSS)  have  a  section  in  which  the  “views  of  others”  can  be 
documented.  The  purpose  is  to  provide  any  differing  views  amongst  various  stakeholders, 
specifically  influential  stakeholders,  when  staffing  a  package.  Bringing  contentious 
viewpoints  to  the  table  early  in  the  process  has  several  benefits.  It  allows  you  to 

1 .  take  the  time  to  grasp  the  heart  of  an  issue  and  what  you  want  to  transmit, 

2.  clearly  articulate  your  position,  and 

3.  clearly  articulate  the  views  of  others. 

Once  this  is  done,  the  package  gets  sent  up  the  chain.  The  structure  of  an  SSS 
allows  for  clear  communications  on  paper  rather  than  dealing  with  the  myriad  information,  or 
often  mis-information  (as  one  respondent  discussed),  that  goes  through  e-mail.  Additionally, 
“if  you  don’t  accept  or  work  those  views  of  others  from  the  get  go,  by  the  time  you  end  up 
briefing  your  leadership,  and  then  your  leadership’s  leadership,  you  end  up  entrenched  in  a 
position  and  you  end  up  entrenched  so  much  that  it’s  hard  to  walk  backwards  from  anymore. 
So  it  removes  your  flexibility  from  a  compromise  or  otherwise.”  Although  it  often  works  out 
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in  the  end,  it  can  be  quite  painful  to  go  back  several  layers  in  the  staffing  process  and  the 
resultant  coordination  change  when  a  relatively  small  or  easy  change  could  have  been 
accomplished,  provided  it  was  worked  up  front. 

When  discussing  how  best  to  communicate  or  “sell”  an  issue,  it  is  very  important  to 
communicate  across  paradigms.  Providing  information  in  a  way  that  program  managers, 
users,  budgeters,  engineers,  and  senior  leaders  in  oversight  all  understand  will  help  prevent 
confusion  and  delays,  particularly  in  the  staffing  process.  Similar  to  knowing  your  audience 
when  giving  a  briefing,  generally  it  is  beneficial  for  a  PM  to  know  the  audience  for  each 
particular  briefing,  meeting,  and  document  and  tailor  the  product  to  the  audience.  A  briefing 
inundated  with  technical  jargon  and  specifications  is  probably  not  best  when  providing 
program  status  to  the  user  or  a  senior  leader. 

Theme  2  Sub-Element  3:  Navigating  Oversight 


Qualitative  Assessment:  5 

Survey  Results: 

Junior: 

Senior: 

Avg  response: 

3.55 

4.06 

%  Unknown: 

19.5% 

N/A 

ANOVA  Significant? 

Yes 

Figure  9.  Overall  Assessment  for  Theme  2  Sub-Element  3 

•  Pick  and  choose  battles  while  preventing  “blood  in  the  water” 

•  Acquisition  oversight  lacks  government  PM  experience 

•  Reduce  oversight  by  executing  the  plan 

•  Smartly  defend  program  budgets 

This  sub-element  discusses  creative  practices  in  working  with  oversight.  Current 
oversight  also  has  several  shortcomings.  To  be  expected,  senior  leaders  had  a  significant 
difference  in  responses  to  the  importance  of  navigating  oversight  than  did  students.  Seniors 
placed  more  emphasis  on  how  to  creatively  navigate  oversight,  especially  the  subset  of 
senior  leaders  with  program  director  experience. 

First,  acquisition  experience  is  lacking  in  acquisition  oversight  positions.  Political 
appointees  often  come  from  industry,  but  as  one  respondent  commented,  “I’ve  been  to  all 
the  schools  you’re  supposed  to,  and  they  always  talk  about  how  industry  does  things. 
Industry  and  government  are  simply  very  different,  and  the  same  approaches  will  not  work 
for  both.”  Respondents  also  noted  that  the  inexperience  results  in  a  lack  of  urgency. 
Techniques  to  work  with  inexperienced  oversight  include  clearly  making  your  case  for  what 
you  are  doing  and  laying  out  when  a  decision  must  be  made  (and  the  rationale  and 
outcomes  if  a  decision  is  not  made  by  then).  If  this  does  not  work,  allies  either  up  the  chain 
or  in  other  oversight  positions  must  be  gained  to  defend  and  promote  your  position.  An 
operations  advocate  at  the  MAJCOM  or  HQ  level  was  cited  as  an  extremely 
beneficial/influential  ally.  Operations  advocates  will  defend  the  program’s  requirements, 
criticality,  and  need  as  the  user,  rather  than  the  program  office  defending  its  own  jobs. 
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Also,  one  way  to  reduce  program  oversight  is  to  reduce  the  ACAT  level  of  the 
program  whenever  possible.  For  example,  ten  $100  million  programs  have  much  fewer 
reporting  requirements  than  one  $1  billion  program.  This  will  allow  each  program  to  be 
smaller  and  leaner,  and  have  less  oversight  (all  else  held  equal).  One  ACAT  1 D  program 
noted  how  the  documentation  requirements  for  a  milestone  review  have  become 
debilitating — 96  documents  containing  12,000+  pages.  As  the  literature  review  showed, 
increasing  a  program’s  classification  level  reduces  oversight  as  well.  However,  both  a 
program’s  classification  and  ACAT  level  are  determined  by  either  law  (for  the  ACAT  level)  or 
strict  policies  (classification  level);  therefore,  a  PM  has  little  authority  to  change  these  after 
program  conception. 

When  navigating  oversight,  PMs  must  pick  and  choose  their  battles  on  the  few 
issues  on  which  they  are  not  willing  to  compromise.  This  will  reinforce  to  the  community 
what  is  not  negotiable  from  the  PM’s  point  of  view.  Correlated  to  this,  one  must  prevent 
“blood  in  the  water”  during  decision  reviews.  This  refers  to  a  stakeholder  or  staff  member 
attacking  controversial  issues  of  the  program  during  a  meeting.  The  PM  must  directly  and 
convincingly  quell  these  arguments  so  that  other  stakeholders  do  not  latch  on,  much  like 
sharks  when  there’s  blood  in  the  water.  For  example,  if  a  stakeholder  questions  the 
reasoning  for  the  contract  type  in  the  acquisition  strategy,  the  PM  should  then  and  there 
explain  why  it  is  the  best  contract  type  and  incentive  structure  for  the  program.  A  hesitant 
answer  or  having  to  get  back  to  the  stakeholder  later  allows  for  other  stakeholders  to  look 
into  the  issue  and  lose  confidence  in  the  PM  having  the  requisite  control  and  understanding 
of  the  program.  Of  course,  this  needs  to  be  tempered  with  difficult,  unforeseen  questions 
that  do  not  have  a  known  answer.  In  these  (hopefully  rare)  cases,  a  PM  should  promise  to 
get  back  to  the  person  as  quickly  as  possible.  In  summary,  keeping  the  “blood  out  of  the 
water”  can  be  immensely  beneficial. 

Practices  in  which  programs  defend  their  budgets  (with  integrity)  reduce  program 
oversight  as  well.  The  best  way  to  defend  against  budget  cuts  and  reduce  intervention  is 
simply  to  stay  green — obligate  and  expend  money  on  time.  Second,  programs  should  make 
every  effort  to  fund  disconnects  internally,  as  no  one  ever  wants  to  ask  for  more  money  (nor 
is  it  currently  available).  The  19.5%  unknown  responses  from  students  in  this  sub-element 
arise  predominantly  from  this  survey  question.  Surprisingly,  40%  of  students  did  not  know  if 
programs  should  fund  disconnects  internally  to  the  maximum  extent  possible.  Therefore,  it 
is  recommended  that  the  appropriate  continuing  education  course  expand  the  teaching  on 
how  PMs  can  avoid  program  interference  by  smartly  managing  funds  internally.  Although 
this  is  of  particular  value  to  program  directors,  PMs  at  all  levels  can  still  learn  from  this 
heuristic  and  do  what  they  can  to  manage  funds  allowing  for  some  degree  of  flexibility. 

Third,  perceptions  are  worse  than  reality  in  many  areas  of  government  acquisitions.  If  a 
program  is  perceived  to  be  fat  (over-funded)  or  behind  schedule,  whether  it  is  true  or  not, 
the  program  is  a  more  apt  candidate  for  cuts. 

Also,  when  hiring  a  material  leader,  some  programs  may  find  it  highly  beneficial  to 
hire  one  with  recent  PEM  experience.  For  example,  a  pre-Milestone  B  program  (even 
though  it  is  not  technically  called  a  program  yet)  will  experience  numerous  decision  reviews, 
staffing,  and  oversight  during  the  Milestone  B  and  source  selection  processes.  Recent  PEM 
experience  greatly  increases  the  process  familiarity  and  likelihood  that  recent  relationships 
will  prove  useful  in  working  the  system. 

Conclusions 

In  review,  the  predominant  finding  of  this  research  is  that  senior  acquisition 
professionals  believe  that  relationships  and  building  trust  are  of  paramount  importance.  A 
high  correlation  exists  between  three  main  sub-elements:  Break  Down  Barriers  and  Build 
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Relationships,  External  Communications  Strategy,  and  Navigating  Oversight.  The  first  two 
are  vital  to  effectively  and  efficiently  navigating  oversight.  Both  program  directors  and 
respondents  with  rapid  experience  chose  these  three  sub-elements  as  their  top  three 
responses. 

For  Navigating  Oversight,  program  directors  more  often  accepted  the  level  of 
oversight  as  a  fact  of  life,  so  they  work  hard  to  efficiently  work  with  and  through  the  oversight 
for  program  success.  However,  rapid  acquisition  organizations  navigated  the  oversight 
process  by  delegating  approval  authorities  and  tailoring  programs  from  the  start,  thus 
avoiding  a  degree  of  oversight  from  the  beginning. 

Additionally,  junior  personnel  did  not  believe  the  relationships  nor  the  oversight 
aspects  to  be  as  important  as  the  senior  leaders  judged.  Therefore,  an  opportunity  exists 
for  DAU  or  AFIT  classes  to  bolster  the  material  relating  to  these  topics.  This  is  especially 
important  not  only  because  the  senior  leaders  attribute  success  to  these  areas,  but  because 
relationships  can  be  built  over  a  career  and  the  process  of  building  relationships  can  begin 
at  the  start  of  one’s  acquisition  journey. 

Recommendations  for  Future  Research 

The  following  are  recommendations  for  future  research.  Future  research  can  be 
accomplished  to  investigate  the  root  cause  of  the  significant  differences  shown  between 
introductory,  mid-level,  and  senior  acquisition  professionals,  both  for  differences  in  the  Likert 
scale  responses  and  for  questions  with  a  significant  number  of  “unknown”  responses. 
Additionally,  the  same  thesis  methodology  could  be  applied  to  industry  program  managers 
to  assess  the  external  validity  of  this  research  to  industry. 
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Abstract 

As  directed  by  the  National  Defense  Authorization  Act  (NDAA)  of  2010,  Public  Law  111-84, 
the  defense  acquisition  community  is  transitioning  in  an  effort  to  adopt  software  best 
practices  for  delivering  information  technology  in  an  incremental  and  iterative  model.  The 
Deputy  Secretary  of  Defense  provided  a  report  to  Congress  titled  A  New  Approach  for 
Delivering  Information  Technology  Capabilities  in  the  DoD,  delineating  the  overarching 
framework  to  reform  the  acquisition  of  information  technology  to  better  address  and  fulfill 
warfighter  requirements.  Many  governmental  agencies,  anticipating  future  directives,  are 
implementing  Agile  software  development  methodologies  and  demonstrating  success  using 
these  methodologies  on  DoD-sponsored  programs.  As  an  example  of  this,  the  Rapid 
Integration  and  Test  Environment  (RITE)  established  by  SSC  Pacific  in  2008  provides  a 
standardized  Agile  development  environment  for  its  C2  programs.  Much  of  the  work  to  date 
has  addressed  program  items  controlled  at  lower  command  levels  while  awaiting 
restructuring  of  the  acquisition  milestone  and  review  requirements  specified  in  DoDI  5000.02. 
This  report  presents  the  research  completed  in  analyzing  traditional  acquisition  program 
milestone  reviews  and  documentation  requirements  and  identifies  streamlining  opportunities 
that  support  Agile  development.  The  report  also  validates  the  RITE  initiative  in  providing  the 
structured  engineering  approach  that  makes  Agile  development  viable  in  a  DoD  acquisition 
environment. 

Introduction 

The  National  Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2010,  Public  Law 
111-84,  Section  804 — hereafter  referred  to  as  Sec.  804,  2010  NDAA — established  the 
requirement  for  the  Department  of  Defense  (DoD)  to  streamline  the  acquisition  of 
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information  technology.  In  response  to  that  request,  the  Office  of  the  Deputy  Secretary  of 
Defense  (2010)  provided  a  report  titled  A  New  Approach  for  Delivering  Information 
Technology  Capabilities  in  the  DoD.  This  report  created  the  overarching  framework  to 
reform  the  acquisition  of  information  technology  to  better  address  and  fulfill  warfighter 
requirements.  While  this  new  requirement  established  the  basics  for  streamlining  information 
technology  acquisition,  it  did  little  to  provide  meaningful,  actionable  practices  that  an 
acquisition  program  can  execute.  The  goal  of  this  research  was  to  identify  opportunities  to 
create  actionable  Agile  processes  that  information  technology  programs  can  use  to  execute 
streamlined  programs. 

Background 

The  Sec.  804,  2010  NDAA  requirement  established  the  parameters  for  the  new 
acquisition  process  based  on  the  March  2009  report  of  the  Defense  Science  Board  (DSB) 
Task  Force  titled  Department  of  Defense  Policies  and  Procedures  for  the  Acquisition  of 
Information  Technology.  The  report  was  required  to  include  several  characteristics  that 
Congress  determined  necessary  for  successful  implementation: 

1 .  early  and  continual  involvement  of  the  user; 

2.  multiple,  rapidly  executed  increments  or  releases  of  capability; 

3.  early,  successive  prototyping  to  support  an  evolutionary  approach;  and 

4.  a  modular,  open-systems  approach.  (NDAA  for  Fiscal  Year  2010,  2009) 

These  characteristics  are  significant  in  that  they  also  describe  the  elements 
indicative  of  an  Agile  development  methodology. 

In  response  to  Sec.  804,  2010  NDAA,  the  DoD  provided  a  report  to  Congress 
highlighting  its  plans  to  reinvent  the  IT  acquisition  process.  Noting  the  departure  necessary 
from  a  traditional  acquisition  process,  the  DoD  provided  the  following: 

Acquisition  activities  in  the  new  process  for  delivering  IT  capability  will  differ 
significantly  from  the  traditional  weapon  system  development  acquisition  process  and  will  be 
separately  defined  in  DoD  IT  acquisition  policy  issuances.  The  IT  acquisition  process  will  be 
agile  to  respond  to  a  dynamic  technology  environment  and  to  address  unique  challenges, 
such  as  cyber  threats  (Office  of  the  Deputy  Secretary  of  Defense,  2010,  p.  9). 

As  shown  in  the  next  section,  this  approach  provides  a  flexible  structure  dedicated  to 
positive,  customer-driven  outcomes. 

Agile  Development 

Agile  development  focuses  on  close  customer  interaction  and  rapid,  iterative,  and 
incremental  development  cycles  that  produce  a  working  product.  This  approach  focuses  on 
early  feedback  and  flexibility  adapting  to  customer  needs. 

In  describing  Agile  methods,  Lapham  et  al.  (201 1 )  noted  that  the  concepts  and 
practices  associated  with  Agile  development  arose  out  of  the  Agile  Alliance.  In  an  effort  to 
identify  an  alternative  to  elaborate  and  time-consuming  software  development  processes, 
the  Agile  Alliance  created  a  set  of  values  that  focus  on  people,  collaboration,  and 
development  of  quality  software  products  for  their  customers  (Lapham  et  al.,  201 1 ,  p.  1 ). 

The  Agile  Alliance’s  efforts  resulted  in  the  Agile  Manifesto  for  Agile  Software 
Development: 

We  are  uncovering  better  ways  of  developing  software  by  doing  it  and  helping 
others  do  it.  Through  this  work  we  have  come  to  value: 
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Individuals  and  interactions  over  processes  and  tools 
Working  software  over  comprehensive  documentation 
Customer  collaboration  over  contract  negotiation 
Responding  to  change  over  following  a  plan 
That  is,  while  there  is  value  in  the  items  on  the  right,  we  value  the  items  on 
the  left  more.  (Lapham  et  al. ,  201 1 ,  p.  1  )1 

Critics  of  Agile  development  cite  documentation  reduction  as  problematic  in 
development  efforts,  but  these  concerns  are  discounted  by  seasoned  developers.  In  Agile 
development,  the  amount  of  documentation  is  determined  by  the  software,  not  the  desire  of 
the  developer.  It  is  essential  to  understand  that  while  documentation  is  important,  it  should 
not  act  as  a  replacement  for  communication  and  collaboration.  Regarding  Agile 
development’s  approach  to  documentation,  Lapham,  Williams,  Hammons,  Burton,  and 
Schenker  (2010)  observed,  “The  Agile  community  would  argue  instead  that  documentation 
is  important,  but  no  more  documentation  should  be  created  than  is  absolutely  necessary  to 
support  the  development  itself  and  future  sustainment  activities”  (p.  4).  Documentation 
developed  using  the  Agile  methodology  can  support  the  intent  and  objectives  of  the 
documentation  requirements  of  the  DoD  acquisition  process. 

Agile  development  is  not  the  only  initiative  working  to  streamline  and  improve  the 
effectiveness  of  development  activities.  The  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR)  Rapid  Integration  and  Test  Environment  (RITE)  initiative  focused  their  efforts  on 
key  areas  in  the  development  cycle  that  work  collectively  to  shorten  cycle-time  and  improve 
the  efficiency  of  the  development  effort. 

Rapid  Integration  and  Test  Environment 

In  2008,  the  Program  Executive  Office  (PEO)  Command,  Control,  Communications, 
Computers,  and  Intelligence  (C4I),  Command  and  Control  Program  Office  (PMW  150) 
began  implementation  of  the  RITE  initiative.  This  initiative  was  born  out  of  necessity  in  that 
the  existing  process  for  requirements  definition  and  management,  as  well  as  processes  for 
software  development,  did  not  consistently  deliver  high-quality  Navy  Command  and  Control 
(C2)  systems  either  on  time  or  within  budget. 

The  RITE  initiative,  as  implemented,  represents  a  new  life  cycle  model  for  Navy  C2 
software  that  meets  many  of  the  process  objectives  identified  in  Sec.  804,  2010  NDAA  and 
improves  efficiencies  in  Navy  C2  application  development.  RITE  places  increased  emphasis 
on  early  and  frequent  customer  interaction  and  software  testing,  as  well  as  necessary 
software  engineering  practices  at  the  source  code  level.  RITE  is  a  structured  approach  to 
software  development,  taking  full  advantage  of  technology  advances  and  open-source 
models  to  automate  processes  and  shorten  development  cycles — thereby  increasing  the 
maintainability  of  the  software  baselines.  The  new  automated  processes  also  allow  a 
reduction  in  low-value-added  processes  and  manually  developed  reports,  further 
streamlining  the  acquisition  cycle  and  improving  efficiencies.  The  initiative  clarifies  software 
delivery  requirements,  adds  additional  engineering  rigor  to  deliverables,  and  reduces  the 
opportunity  for  misunderstanding  between  end  users  and  developers.  Lastly,  RITE  uses  a 
centralized  information  repository  that  allows  all  stakeholders  to  communicate,  coordinate, 
and  collaborate  virtually. 


1  The  Manifesto  for  Agile  Development  was  created  during  a  meeting  of  representatives  from  across 
the  nascent  Agile  community  and  included  the  following:  Kent  Beck,  Mike  Beedle,  Arie  van 
Bennekum,  Alistair  Cockburn,  Ward  Cunningham,  Martin  Fowler,  James  Grenning,  Jim  Highsmith, 
Andrew  Hunt,  Ron  Jeffries,  Jon  Kern,  Brian  Marick,  Robert  C.  Martin,  Steve  Mellor,  Ken  Schwaber, 
Jeff  Sutherland,  and  Dave  Thomas. 
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As  RITE  has  evolved  and  process  improvements  have  been  realized,  additional  uses 
for  RITE  in  support  of  the  C2  life  cycle  have  been  identified.  This  support  includes  facilitating 
close  collaboration  with  outside  agencies  to  ensure  that  the  development  knowledge  and 
test  and  evaluation  (T&E)  results  are  shared  in  order  to  reduce  overall  project  time.  Figure  1 
shows  the  RITE  processes  as  they  align  with  all  four  phases  of  the  new  IT  acquisition  life 
cycle.  The  arrows  indicate  areas  where  RITE  (consisting  of  people,  processes,  and 
infrastructure)  directly  supports  the  acquisition  of  Navy  C2  capabilities  and  systems. 


Figure  1.  RITE  Alignment  With  2010  IT  Acquisition  Changes 


Defense  Acquisition  Management  System 

The  Defense  Acquisition  Management  System  (see  Figure  2)  is  the  management 
process  guiding  all  DoD  acquisition  programs.  The  initiating  directive,  DoD  Directive  (DoDD) 
5000.01,  provides  the  policies  and  principles  that  govern  the  defense  acquisition  system, 
and  DoD  Instruction  (DoDI)  5000.02,  Operation  of  the  Defense  Acquisition  System,  provides 
the  management  framework  that  implements  these  policies  and  principles.  “The  Defense 
Acquisition  Management  Framework  provides  an  event-based  process  where  acquisition 
programs  progress  through  a  series  of  milestones  associated  with  significant  program 
phases”  (DoD,  2012). 

The  Defense  Acquisition  Management  System  is  used  throughout  the  DoD  as  the 
single  overarching  methodology  for  acquiring  business  and  weapons  systems. 
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User  Needs 


Technology  Opportunities  &  Resources 


•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition 
management  system 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 
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Figure  2.  The  Defense  Acquisition  Management  System 
Related  Research 


Defense  Science  Board  Task  Force  Report  on  Department  of  Defense  Policies  and 
Procedures  for  the  Acquisition  of  Information  Technology 

In  March  2009,  the  DSB  Task  Force  reported  on  the  evaluation  of  the  acquisition  of 
information  technology  (IT)  within  the  DoD.  This  report  identified  critical  problems  with  the 
management  of  IT  acquisitions  using  an  enterprise  approach  resulting  in  a  “profound 
operational  impact”  (DSB  Task  Force,  2009,  p.  1).  The  report  identified  problems  in 
responsiveness  and  the  ability  to  address  operational  needs.  Citing  a  2006  DSB  study  titled 
Information  Management  for  Net  Centric  Operations,  the  report  noted, 

Especially  important,  according  to  the  2006  report,  was  that  much  of  the 
military  capability  used  to  support  the  conflicts  was  paid  with  supplemental 
funding — programs  that  were  not  part  of  the  Department’s  planned  capability. 
This  circumstance  reflects  the  fact  that  the  need  for  such  programs  could  not 
be  predicted  during  previous  core  program  and  budget  planning,  and  the 
system  was  not  sufficiently  agile  to  react  once  the  need  was  apparent.  (DSB 
Task  Force,  2009,  pp.  1-2) 

The  report  goes  on  to  identify  the  evolution  of  weapons  system  software  reliance  in 
the  1970s  at  20%  to  as  much  as  80%  in  2000.  This  is  a  critical  issue  in  light  of  the  reduction 
in  U.S.  computing  graduates  and  qualified  expert  government  staff  and  increased  reliance 
on  IT  at  a  time  of  rising  vulnerabilities  and  threats  (see  Figure  3;  DSB  Task  Force,  2009,  p. 
6). 
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Figure  3.  The  Perfect  IT  Storm 

(DSB  Task  Force,  2009) 


The  DSB  Task  Force’s  findings  identified  the  need  for  a  unique  acquisition  process 
for  IT.  Commenting  on  the  failure  of  major  defense  systems,  the  task  force  also  identified 
the  need  to  shorten  the  lengthy  acquisition  process  and  to  provide  the  flexibilities  necessary 
to  support  continuous  changes  and  upgrades.  Other  critical  elements  of  change  identified 
by  the  DSB  Task  Force  include  the  need  to  align  acquisition  authorities  and  organizational 
structure  under  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  (OUSD[AT&Lj)  to  better  manage  the  technical  aspects  of  IT  acquisitions  and 
the  need  to  consider  proven  experience  as  an  added  component  in  evaluating  the  education 
and  certification  of  members  of  the  acquisition  workforce. 

Considerations  for  Using  Agile  in  DoD  Acquisition  (Carnegie  Mellon  University, 
Software  Engineering  Institute) 

This  document  was  created  to  provide  additional  information  on  Agile  development 
as  it  relates  to  DoD  acquisitions,  references  actual  DoD  programs  that  have  benefited  from 
the  adoption  of  Agile  practices  within  their  respective  programs,  and  includes  analysis  of 
relevant  literature  regarding  Agile  development.  Lapham  et  al.  (2010)  answered  many 
questions  regarding  Agile  development,  but  they  specifically  answered  whether  Agile 
development  methods  are  able  to  produce  better  products  within  cost  and  schedule 
requirements  (yes)  and  addressed  the  barriers  which  inhibit  the  DoD’s  adoption  of  Agile 
development  methods. 

In  determining  the  barriers  to  DoD’s  Agile  development  adoption,  Lapham  et  al. 
(2010)  noted, 

The  barriers  to  adopting  Agile  in  the  DoD  appear  to  be  primarily  cultural.  That 
is  to  say  that  there  is  little  in  the  way  of  regulation  or  guidance  provided  in 
DoDI  5000.02  that  would  prevent  the  use  of  Agile.  This  instruction  does 
impose  specific  constraints  on  the  acquisition  office,  but  these  constraints 
would  be  true  of  any  development  environment,  (p.  27) 

While  not  finding  any  primary  barriers  within  the  DoDI  5000.02,  Lapham  et  al.  (2010) 
did  address  issues  with  the  Federal  Acquisition  Regulation,  citing  the  need  to  address 
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contracting  requirements  to  support  Agile  development.  These  changes  would  require  the 
accommodation  of  Agile  as  part  of  a  system’s  acquisition  strategy  at  the  beginning  of  a 
program  development  effort  (Lapham  et  al. ,  2010,  p.  27).  The  authors  also  pointed  to 
significant  concerns  regarding  milestone  reviews  within  the  DoD  acquisition  system: 

A  very  specific  acquisition  issue  and  sticking  point  is  that  Agile  methodology 
does  not  accommodate  large  capstone  events  such  as  Critical  Design 
Review  (CDR),  which  is  usually  a  major,  multi-day  event  with  many  smaller 
technical  meetings  leading  up  to  it.  This  approach  requires  a  great  deal  of 
documentation  and  many  technical  reviews  by  the  contractor.  (Lapham  et  al., 
2010,  p.  13) 

In  addressing  the  primary  questions  raised  regarding  Agile  development  and  its  use 
within  the  DoD,  Lapham  et  al.  (2010)  noted  that  end-user  participation  and  culture  are 
issues  that  must  be  addressed  before  using  Agile  methods  within  a  program  (p.  44). 

Agile  Methods:  Selected  DoD  Management  and  Acquisition  Concerns  (Carnegie 
Mellon  University,  Software  Engineering  Institute) 

This  document  is  the  second  in  a  series  regarding  Agile  development  methods  and 
the  use  of  Agile  within  the  DoD.  While  focusing  on  a  better  understanding  of  Agile 
development  as  it  pertains  to  the  DoD  acquisition  system,  Lapham  et  al.  (2011)  targeted  this 
report  to  address  Agile  development  implementation  approaches  for  acquisition  and 
development  personnel  (p.  2). 

Lapham  et  al.  (2011)  provided  thorough  discussions  of  Agile  development,  why  Agile 
methods  are  increasing  within  the  DoD,  contracting  requirements  for  implementation  within 
Agile  programs,  and  the  use  of  change  management  within  an  organization,  specifically 
applicable  to  a  program  management  office  (PMO),  to  implement  Agile  methods.  Most 
applicable  to  the  analysis  within  this  paper  is  the  discussion  of  milestone  reviews  within 
systems  development  and  its  effect  on  Agile  development.  (Lapham  et  al.,  201 1 ,  pp.  10-11). 
The  authors  provided  a  thorough  evaluation  of  milestone  reviews,  including  the  effort 
required  to  produce  the  supporting  documentation  and  not  the  challenges  associated  with 
adapting  a  program’s  milestone  reviews  to  an  Agile  methodology: 

The  intent  of  any  technical  milestone  review  is  for  evaluation  of  progress 
and/or  technical  solution.  For  PMOs  trained  and  experienced  in  the  traditional 
acquisition  methods,  evaluating  program  progress  and  technical  solutions 
follows  well  established  guidelines  and  regulations.  Very  specific 
documentation  is  produced  to  provide  the  data  required  to  meet  the  intent  of 
the  technical  review  as  called  out  in  the  program  specific  Contract  Data 
Requirements  List  (CDRL).  The  content  of  these  documents  and  the  entry 
and  exit  criteria  for  each  review  is  well  documented.  However,  even  in 
traditional  acquisitions  (using  traditional  methods),  these  documents,  exit  and 
entry  criteria  can  be  and  usually  are  tailored  for  the  specific  program.  Since 
the  documentation  output  from  Agile  methods  appears  to  be  “light”  in 
comparison  to  traditional  programs,  the  tailoring  aspects  take  on  additional 
aspects.  Some  of  the  specific  challenges  for  Agile  adoption  that  we  observed 
during  our  interviews  that  must  be  addressed  are  as  follows: 

•  incentives  to  collaborate, 

•  shared  understanding  of  definitions/key  concepts, 

•  document  content — the  look  and  feel  may  be  different  but  the  intent  is 
the  same — and 
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•  regulatory  language.  (Lapham  et  al.,  201 1 ,  pp.  38-39) 

Analytical  Approach 

The  analytical  approach  involved  exhaustive  analysis  of  technical  reviews  and 
documentation  to  identify  possible  areas  in  which  duplication  or  overlap  currently  exists 
within  the  review  structure  or  the  documentation  set  required  when  developing  a  product. 

The  review  included  a  thorough  analysis  of  all  milestone  reviews  and  documentation 
associated  with  a  typical  development  effort.  The  analysis  examined  the  technical  definition 
of  each  review,  the  statutory  or  regulatory  requirement  upon  which  it  is  based,  the  program 
participant/organization  responsible  for  execution  of  the  review,  the  program 
participant/organization  responsible  for  conducting  the  review/completing  the  document 
(subordinate  organization — typically  Software  Support  Activity  [SSA],  In-service  Engineering 
Agent  [ISEA],  etc.),  key  team  members  involved,  entrance  and  exit  criteria  for  the  review, 
recipient  of  the  completed  review  results  (PEO,  Milestone  Decision  Authority  [MDA],  etc.), 
any  other  stakeholders,  and  previous  and  next  process  flow  steps.  The  review  process  was 
refined  to  focus  on  the  following  milestone  reviews:  Preliminary  Design  Review  (PDR), 
Critical  Design  Review  (CDR),  Test  Readiness  Review  (TRR),  System  Verification  Review 
(SVR),  and  Production  Readiness  Review  (PRR),  which  were  evaluated  against  Agile 
development  requirements.  Further  analysis  was  conducted  against  the  DoD  and  SPAWAR 
Systems  Command  (SPAWARSYSCOM)  System  Engineering  Technical  Review  (SETR) 
PDR  and  CDR  Risk  Assessment  Checklists  to  provide  a  cross-referenced  analysis  against 
PDR  and  CDR  requirements.  These  checklists  were  targeted  due  to  their  complexity  (The 
DoD  PDR  checklist  is  860  line  items,  and  the  DoD  CDR  checklist  is  929  line  items)  and  their 
applicability  within  development  timelines  associated  with  Agile  development.  Although 
SPAWARSYSCOM  SETR  checklists  for  PDR/CDR  closely  follow  the  DoD  checklists  (with 
871  and  906  line  items,  respectively),  the  difference  in  line  items  represents  tailoring  to 
address  Navy  specific  requirements. 

The  documentation  analysis  included  an  evaluation  of  which  milestones  within  the 
defense  acquisition  system  required  completion  or  updating  of  each  specific  document. 
Additionally,  the  evaluation  included  the  review  of  the  documentation  set  required  by  the 
SPAWARSYSCOM  SETR  Risk  Assessment  Checklists. 

Results 

This  section  highlights  the  pertinent  analysis  of  the  reviews  and  documentation 
information  collected  during  the  preliminary  part  of  this  effort.  Discussions  with  experienced 
program  professionals  and  other  acquisition  workforce  personnel  also  occurred  during  the 
data  collection  and  analysis  phases  to  better  inform  the  group’s  decision-making  process. 

Of  note,  during  the  analytical  phase  of  this  effort,  discussions  regarding  the  role  of 
the  cognizant  technical  authority  (TA)  and  their  impact  (positively  or  negatively)  on  the 
viability  of  the  development  effort.  According  to  the  Naval  Warfare  Systems  Certification 
Policy,  a  TA’s  role  within  an  organization  is  as  follows: 

The  entity  with  the  authority,  responsibility,  accountability,  and  technical 
integrity  to  establish,  monitor,  and  approve  technical  standards,  tools,  and 
processes  in  compliance  with  applicable  DoD  and  DoN  policy,  requirements, 
architectures,  and  standards.  (DoN,  2012,  pp.  B-6) 

While  the  TA’s  role  is  focused  on  institutional  level  technical  compliance,  the  TA’s 
role  remains  secondary  to  the  program  manager’s  (PM’s)  and  MDA’s  role  in  validating  and 
approving  the  planned  milestone  review  and  programmatic  documentation  streamlining 
efforts.  Even  so,  the  TA’s  role  as  the  technical  advocate  in  support  of  development  methods 

ACQUISITION  RESEARCH  PROGRAM: 

CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  194- 


such  as  Agile  cannot  be  overstated.  A  TA’s  commitment  (and  through  extension,  a 
command’s  commitment)  to  Agile  development  can  be  helpful  in  supporting  the  MDA’s 
decision  to  approve  a  PM’s  request  to  eliminate  or  otherwise  minimize  documentation 
requirements. 


Primary  Review  Analysis 

The  initial  analysis  of  technical  reviews  included  the  following:  Initial  Technical 
Review  (ITR),  Alternative  System  Review  (ASR),  Integrated  Baseline  Review  (IBR),  System 
Requirements  Review  (SRR)  .Technology  Readiness  Assessment  (TRA),  System 
Functional  Review  (SFR),  PDR,  CDR,  TRR,  SVR,  Functional  Configuration  Audit  (FCA), 
PRR,  Operational  Test  Readiness  Review  (OTRR),  Physical  Configuration  Audit  (PCA), 
Integration  Readiness  Review  (IRR),  In  Service  Review  (ISR),  Development  Test  Readiness 
Review  (DTRR),  and  Operational  Test  Readiness  Review  (OTRR).  Although  this  analysis 
was  an  essential  first  step  and  helped  to  visualize  individual  reviews  within  the  context  of  the 
DoD  Acquisition  Management  System  (see  Figure  4  ),  no  major  streamlining  opportunities 
were  identified  in  the  analysis. 
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Figure  4.  System  Engineering  Technical  Reviews  According  to  the  DoD  Acquisition 

Management  System 

In  evaluating  the  reviews  against  Agile  development  principles,  it  was  evident  that  to 
achieve  any  streamlining  within  the  review  process,  the  numerous  review  requirements 
would  need  to  be  downsized  and  re-envisioned  to  address  the  primary  elements  of  the 
existing  reviews.  This  was  preliminarily  documented  in  the  DSB  Task  Force’s  (2009)  report 
Department  of  Defense  Policies  and  Procedures  for  the  Acquisition  of  Information 
Technology  (see  Figure  5).  The  DSB  Task  Force’s  (2009)  recommendation  streamlined  the 
milestone  review  process  to  eliminate  the  complex,  all-encompassing  milestone  reviews  in 
favor  of  more  frequent,  tailored  decision  points  that  enable  a  program  to  identify  problems 
earlier,  which  results  in  more  “robust  and  maintainable  designs”  (pp.  52-53). 
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Figure  5.  New  Acquisition  Process  for  Information  Technology 

(DSB  Task  Force,  2009,  p.  48) 


In  the  context  of  the  primary  milestone  reviews  (PDR,  CDR,  and  SVR/PRR),  a 
nominal  Agile  development  structure  was  created  (see  Figure  6),  providing  increment 
releases  (two-year  cycles)  that  include  service  packs  (six-month  cycles  of  completed 
development  efforts  that  have  the  potential  to  be  forwarded  as  release  candidates).  Within 
each  service  pack  is  a  series  of  sprints,  which  represent  a  standard  form  of  Agile 
development.  This  construct  allows  the  identification  of  a  Build  Review  (BR;  reviews  are 
shown  in  red  in  Figure  6)  at  the  beginning  of  each  service  pack,  which  addresses  elements 
of  the  increment  level  PDR  and  subsequent  CDR;  an  Interim  Progress  Review  (IPR)  at 
Sprint  3  or  4  to  assess  progress  regarding  cost,  schedule,  and  performance  and  evaluate 
the  service  pack  functional  backlog  compared  to  the  current  backlog,  validating  the  detailed 
design  of  the  remaining  sprints;  and  a  Fielding  Review  (FR)  at  the  end  of  the  sprint  cycle. 
These  reviews  throughout  the  sprint/service  pack  cycles  supplant  the  traditional 
PDR/CDR/SVR/PRR  reviews  and  relate  directly  to  the  decision  points  described  in  the  DSB 
Task  Force’s  (2009)  report  to  Congress,  as  shown  in  Figure  5. 2 


2  Service  pack  functional  backlog,  from  an  Agile  development  perspective,  is  a  prioritized  listing  of 
allocated  requirements  (in  Agile  terms,  stories)  determined  at  the  beginning  of  the  sprint  to  be 
sufficient  tasking  to  complete  within  the  sprint  cycle.  The  current  backlog  is  the  amount  of  the  service 
pack  functional  backlog  remaining  within  the  sprint  and  is  used  to  determine  the  progress  against  the 
planned  effort. 
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Figure  6.  Linkage  Between  DoD  Acquisition  Mangaement  System  Reviews  and  Agile 

Development  Reviews 

Given  the  potential  differences  in  the  wide  variety  of  program  development  efforts, 
tailoring  of  the  reviews  to  best  support  the  specific  aspects  of  a  program  is  necessary.  This 
customization  can,  as  indicated  previously,  be  structured  such  that  the  sum  of  the  review 
content  is  equal  to  the  sum  of  the  replaced  reviews. 

Just  as  the  reviews  themselves  are  being  streamlined,  the  supporting  documentation 
should  be  streamlined  to  eliminate  unnecessary  effort. 

Documentation  Analysis 

The  documentation  review  resulted  in  a  comprehensive  analysis  that  provides  a 
high-level  overview  of  acquisition  documentation.  Although  it  was  expected,  the  review 
verified  that  because  a  program  is  required  to  increase  reporting  responsibilities  to  address 
statutory  and  regulatory  requirements,  opportunities  for  significant  streamlining  are  greatly 
reduced.  This  is  particularly  true  for  Major  Defense  Acquisition  Programs  (MDAPs)  and 
Major  Automated  Information  Systems  (MAISs).  It  is  the  remaining  programs  that  can 
benefit  from  a  reduction  in  documentation  associated  with  regulatory  requirements; 
specifically,  small  software  intensive  development  efforts.  This  does  not  preclude  the  use  of 
Agile  development  as  a  component  of  larger  projects  (such  as  for  a  software  development 
effort  ancillary  to  a  major  hardware  development  effort),  but  it  will  require  a  significant 
amount  of  negotiation  with  the  MDA. 

In  analyzing  individual  document  requirements,  it  was  apparent  that  aggregate 
generalizations  regarding  documentation  do  little  to  support  the  tailoring  of  a  program  to 
streamline  reporting  requirements  other  than  to  say  that  it  is  possible.  As  Lapham  et  al. 
(2010)  reported, 

Those  programs  that  have  used  Agile  in  software  development  have  found 
that  the  DoD  5000  series  has  great  flexibility  and  does  not  in  fact  preclude  the 
use  of  Agile.  It  appears  that  with  careful  review  and  some  tailoring  an 
alternate  interpretation  can  be  created  so  that  Agile  can  be  used  on  DoD 
programs,  (p.  13) 

This  analysis,  while  correct  in  identifying  the  DoD  5000  series  as  the  prime  set  of 
regulatory  hurdles  with  which  to  contend,  shows  that  a  program  must  also  deal  with 
additional  statutory  and  other  regulatory  requirements  tied  to  acquisition  development.  Even 
if  Service-specific  requirements  (Secretary  of  the  Navy  instructions,  Army  regulations,  etc.) 
and  Defense  Acquisition  Guidebook  requirements  are  removed,  several  Title  10 
requirements  and  other  regulatory  requirements  remain  (such  as  Chairman  of  the  Joint 
Chiefs  of  Staff  Instruction  [CJCSI]  3010.02B,  3100.01A,  3170.01H,  3312.01A,  6212.01D, 
and  8501 .01  A;  DoDD  7045.20;  DoDI  4650.01 , 6055.1 ,  and  7041 .3;  and  Statement  of 
Federal  Financial  Accounting  Standards  [SFFAS]  No.  23). 

The  statutory/regulatory  documentation  breakout  resulted  in  further  decomposition  to 
identify  value-added  versus  negligible-value  or  no-value-added  documentation  (this  was  a 
qualitative  evaluation  associated  nominally  with  a  generic  Agile  software  development 
effort).  Many  documentation  requirements  have  little  or  no  value  in  supporting  a  software 
development  effort  or  the  eventual  fielding  of  software  (such  as  Programmatic 
Environmental,  Safety,  and  Occupational  Health  Evaluation,  Non-Destructive  Test  Plan,  and 
Unique  Identification  Implementation  Plan,  Failure  Modes  Effects  Criticality  Analysis, 
Performance  Based  Logistics  Business  Case  Analysis,  and  Diminishing  Manufacturing 
Sources  and  Material  Shortages);  in  these  cases,  the  PM  should  negotiate  with  the  MDA  to 
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remove  or  reduce  the  documentation  requirement,  as  appropriate.  There  are  many  cases  in 
which  the  value  of  the  document  to  the  development  effort  is  obvious,  and  program 
management  offices  should  identify  those  documents  early  in  the  program  initiation  phase  to 
ensure  proper  planning  to  accommodate  the  necessary  documentation  effort. 

A  program’s  milestone  reviews  and  documentation  streamlining  effort  can  support  a 
project’s  Agile  development;  however,  gaining  MDA  approval  for  those  efforts  can  be 
problematic  without  some  assurance  that  programs  are  still  producing  a  quality  product. 

RITE  provides  many  of  the  necessary  assurances  that  programs  need  to  gain  MDA 
approval. 

RITE  Analysis 

As  described  in  the  background  section,  the  RITE  initiative  was  created  out  of  a  need 
to  improve  the  ability  of  programs  to  meet  cost,  schedule,  and  performance  targets  of  their 
sponsors.  In  adapting  to  the  needs  of  Sec.  804,  201 0  NDAA,  RITE  answers  many  of  the 
concerns  of  PMs  and  MDAs  regarding  the  rigor  necessary  to  successfully  implement  an 
Agile  development  methodology. 

In  following  the  RITE  process,  programs  use  the  RITE  Pillars  (see  Figure  7)  to  guide 
their  efforts  in  supporting  an  Agile  development  effort.  RITE  focuses  a  program’s  efforts  on 
critical  areas  proven  to  be  essential  in  successfully  developing  and  fielding  software 
products  within  cost,  schedule,  and  performance  constraints. 


Figure  7.  RITE  Pillars 

The  RITE  process  is  not,  nor  is  it  intended  to  be,  a  panacea  for  a  program  struggling 
with  Agile  development.  It  is  intended  to  support  Agile  development  and  other  simplified, 
rapid  development  techniques  that  focus  on  product  quality  and  efficient  development. 
Combining  Agile  development  with  RITE  provides  a  program  with  the  structured  engineering 
practices  necessary  for  defense  acquisitions.  The  RITE  focus  on  contracts  is  supported  by 
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Lapham  et  al.  (2011)  in  analyzing  contracting  issues  associated  with  Agile  development: 
“Due  to  the  iterative  nature  of  Agile  and  its  propensity  to  accept  (even  welcome)  change, 
many  contracting  vehicles  present  unique  challenges  for  employing  Agile  methods.  A 
particular  issue  is  the  reporting  and  milestone  requirements  often  levied  against  DoD 
contracts”  (p.  33). 

RITE  also  includes  focus  areas  for  processes,  infrastructure,  and  organization,  which 
provide  necessary  supporting  elements  that  give  Agile  development  structure  without 
becoming  cumbersome  to  the  development  effort.  The  Process  component  of  RITE  puts  a 
greater  level  of  rigor  in  the  development  effort  and  provides  the  structure  necessary  to  keep 
Agile  development  methods  on  track.  The  Infrastructure  component  of  RITE  provides  the 
tools  necessary  to  support  Agile  development  without  hindering  flexibility;  automating  as 
much  of  the  mundane  record-keeping,  configuration  management,  and  test  tools  and  data 
ensures  that  the  development  team  stays  focused  on  development  and  not  on  writing 
reports  and  tracking  software  baselines.  The  Organization  component  of  RITE  focuses  on 
the  teaming  nature  required  in  an  Agile  development  environment.  While  it  is  common  to 
have  a  software  effort  completely  developed  by  a  contractor,  the  RITE  process  has 
identified  key  areas  in  which  government  personnel  support  development  by  integrating 
users,  developers,  and  the  integration/test  team  throughout  the  development  cycle. 

Recommendations 

Although  the  DoD  response  to  the  congressional  requirement  to  reform  the  IT 
acquisition  system  referenced  all  the  key  components  necessary  to  compel  program 
management  offices  to  consider  Agile  development  methods,  little  is  actionable  from  the 
response.  The  DoD  must  focus  efforts  on  adapting  the  DoD  5000  series  to  address 
streamlined  development  methods  and  provide  the  regulatory  authority  to  reduce 
documentation  complexity  while  maintaining  appropriate  oversight.  Pending  a  significant 
change  to  the  DoD  5000  series,  PMOs  can  still  execute  Agile  development — but  not  without 
addressing  milestone  reviews,  contracting,  and  documentation. 

The  milestone  review  process  must  transition  from  monolithic,  all-encompassing 
reviews  to  smaller,  frequent  decision  reviews  focused  on  meeting  development  targets. 
Ensuring  flexibility  in  the  process,  the  reviews  must  accommodate  changing  requirements 
and  quality  development.  The  Office  of  the  Deputy  Secretary  of  Defense  (2010)  report  to 
Congress  provides  the  basic  authority  to  execute  IT  programs  based  on  this  approach  (pp. 
9-14).  The  transition  to  frequent  decision  reviews  must  also  be  accompanied  by  a 
streamlined  documentation  effort. 

Maintaining  the  comprehensive  documentation  requirements  of  a  standard 
acquisition  program  would  severely  reduce  the  value  of  an  Agile  development. 
Documentation  should  be  focused  primarily  on  meeting  the  requirements  of  the 
development  and  sustainment  effort.  Secondary  requirements  should  include  statutory 
documentation  and  regulatory  documentation  that  cannot  be  negotiated  away.  This 
negotiation  with  the  MDA  must  be  executed  as  early  as  possible  in  the  program  initiation 
phase  as  soon  as  documentation  requirements  are  locked  down. 

Where  statutory  and  regulatory  compliance  drives  requirements  outside  the  Agile 
development  structure,  PMOs  should  ensure  that  contracts  address  those  elements  while 
maximizing  the  flexibility  necessary  to  keep  Agile  development  as  the  primary  criteria  upon 
which  the  contract  is  evaluated.  As  Lapham  et  al.  (201 1 )  noted  in  their  assessment  of  the 
value  of  implementing  an  Agile  development  methodology  to  a  PMO,  engagement  above 
the  PMO  level  is  necessary  (including  the  need  for  waivers,  mainly  from  the  MDA)  to 
address  the  departure  from  DoDI  5000.02  requirements: 
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For  example,  a  PMO  that  embraces  the  Agile  principle  that  values  operating 
code  over  extensive  documentation  may  require  a  different  set  of  CDRLs 
when  formulating  a  contract.  This  not  only  requires  a  change  in  perspective, 
but  also  the  creation  of  appropriate  governance  models,  via  tailoring  DoD 
5000.02  and  CDRLs  from  such  events  as  SRR,  PDR,  CDR,  etc.  The  PMO 
involved  may  have  to  seek  waivers  from  higher  up  the  acquisition  chain,  and 
these  higher-ups  must  also  understand  Agile  methods  if  they  are  to 
understand  what  they  are  waiving.  One  of  our  reviewers  cited  a  recent 
contract  using  Agile  methods,  in  which  they  were  bounded  by  an  SDR 
milestone,  but  obtained  approval  to  have  IDRs  (Incremental  Design  Reviews) 
beyond  that  time  instead  of  the  traditional  PDR  and  CDR  cycle,  (p.  24) 

PMOs  supporting  an  Agile  development  effort  must  work  closely  with  their  respective 
TA  to  identify  and  plan  a  successful  acquisition  strategy  that  leverages  the  best  of  Agile 
methods  while  maintaining  the  oversight  necessary  to  ensure  that  a  quality  product  is 
delivered  within  cost,  schedule,  and  performance  parameters.  The  PM  and  TA  must  present 
a  unified  front  in  gaining  approval  from  the  MDA.  The  TA,  providing  the  institutional  backing 
for  Agile  development,  should  champion  the  effort,  while  the  PM  provides  program  specific 
details  that  support  the  program’s  streamlining  requests. 

This  interaction  between  the  PM  and  MDA  is  essential  to  the  success  of  any  Agile 
development  effort  absent  significant  changes  to  current  acquisition  regulations  to  address 
the  Sec.  804,  2010  NDAA  requirements.  Implementation  of  RITE,  within  the  context  of  an 
acquisition  program’s  Agile  development  effort,  will  assist  PMOs  in  validating  and  ensuring 
compliance  with  critical  acquisition  elements,  which  is  essential  to  garner  the  support  of  the 
MDA.  RITE  is  an  Agile  enabler  for  the  government. 

Conclusion 

The  analysis  regarding  the  effort  necessary  to  streamline  a  program’s  milestone 
reviews  and  documentation  requirements  confirm  previous  research  regarding  the 
applicability  of  Agile  development  within  a  DoD  acquisition  environment.  These  results 
require  an  up-front  investment  in  time  and  effort  to  produce  a  meaningful  reduction  in  the 
milestone  review  and  documentation  effort.  PM  engagement  with  the  MDA,  in  concert  with 
the  TA,  is  essential  in  gaining  the  approvals  necessary  to  support  Agile  development.  The 
use  of  the  RITE  process  supports  the  PM’s  objective  of  creating  a  structured  environment 
that  remains  conducive  to  Agile  development  and  provides  the  MDA  with  the  comfort  level 
needed  for  approval  of  a  streamlined  milestone  review  and  documentation  effort. 
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Abstract 

Budget  reductions  will  require  the  Department  of  Defense  (DoD)  to  make  difficult  decisions 
on  how  to  invest  limited  resources  and  make  current  programs  more  affordable.  Traditional 
acquisition  methods  are  lengthy,  serial,  gate-like  processes,  built  around  stringent 
specifications  and  arms-length  relationships.  By  contrast,  Challenge-Based  Acquisition 
(ChBA)  utilizes  transparent,  accessible,  concrete  challenges  to  satisfy  warfighter  needs  and 
stimulate  industry  innovation.  Challenges  enable  DoD  programs  to  assess  actual 
performance  against  clearly  defined  mission  objectives  and  create  incentives  for  industry  to 
innovate.  ChBA  thus  offers  a  more  transparent  approach  to  fielding  new  capabilities, 
upgrades,  and  enhancements  to  existing  systems. 

Mandate  for  Change 

It’s  time  to  fundamentally  change  the  way  that  we  do  business  in  Washington.  To 
help  build  a  new  foundation  for  the  21st  century,  we  need  to  reform  our  government  so  that 
it  is  more  efficient,  more  transparent,  and  more  creative.  That  will  demand  new  thinking  and 

a  new  sense  of  responsibility  for  every  dollar  that  is  spent. 

-  President  Barack  Obama  (2009) 

Fewer  than  half  of  the  programs  in  the  Department  of  Defense  (DoD)  Major  Defense 
Acquisition  portfolio  have  met  established  metrics  for  cost  or  performance  (GAO,  2011a). 
Even  worse,  the  DoD  has  canceled  entire  programs  for  cost  overruns  under  the  Nunn- 
McCurdy  Amendment  after  investing  billions  of  dollars  that  could  have  been  used  elsewhere 
across  the  department  (GAO,  2011b).  According  to  the  Government  Accountability  Office 
(GAO),  50  of  74  breaches  involved  engineering  design  issues  discovered  after  production 
had  begun. 

Traditional  DoD  acquisition  follows  a  lengthy,  serial  process  based  upon  a  plethora 
of  documentation  as  required  by  the  DoD  5000  Series  of  Instructions  and  Directives  (DoD, 
2003b)  as  well  as  numerous  Service-specific  acquisition  guidelines.  In  these  documents, 
mission  needs  become  program  requirements,  which  are  then  quantified  as  performance 
parameters,  defined  as  system  attributes,  tracked  through  derived  technical  performance 
measures,  and  included  in  a  government/industry  exchange  of  system  specifications.  Along 
this  serial  path,  the  linkage  of  program  requirements  to  mission  performance  typically 
becomes  unclear  and  often  inaccurate.  Alternatively,  in  some  cases,  system  specifications 
become  far  too  rigid  and  detailed,  thus  stifling  opportunities  for  innovation.  Despite  best 
efforts  by  programs  to  mitigate  risk  through  verification  and  validation  using  the  systems 
engineering  process,  even  a  perfectly  executed  program  can  still  produce  a  quality  product 
that  is  often  “late  to  the  fight,”  operationally  ineffective,  or  unsuitable  even  if  it  addresses  the 
original  mission  need. 

Furthermore,  most  contracts  are  awarded  using  government  source  selection 
evaluations  based  on  industry  paper  proposals  rather  than  “actual”  product  performance. 
This  creates  an  incentive  for  industry  to  produce  flawless  documents  with  highly  optimistic 
cost,  schedule,  and  performance  projections  that  meet  or  exceed  every  requirement  in  the 
government’s  request.  As  a  result,  performance  during  program  execution  often  falls  short  of 
the  government’s  expectations  and  cost  and  schedule  overruns  become  nearly  inevitable. 
These  unrealistic  proposals  become  particularly  problematic  when  there  is  little  prospect  for 
additional  competition  throughout  the  acquisition  life  cycle,  which  may  lock  the  program  into 
a  single  solution  and  provider. 

The  resulting  disappointment  creates  an  arms-length  relationship  between  the 
contractor  and  the  government,  limiting  trust,  communication,  and  transparency.  This  can  be 
particularly  problematic  given  the  long  life  cycle  of  many  defense  acquisition  programs.  The 
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impact  of  this  tense  relationship  can  raise  costs  related  to  bidding  and  negotiating  contracts 
and  slow  the  process  of  coming  to  acceptable  terms  and  conditions  (Crook,  Ketchen, 

Combs,  &  Patterson,  2012).  For  example,  a  recent  study  concluded  that  the  DoD  currently 
spends  roughly  $400  billion  each  year  acquiring  products  and  services  from  its  contractors, 
with  about  $100  billion  of  that  amount  spent  on  administrative  costs  alone.  By  cutting 
unneeded  bureaucracy,  defense  officials  could  reduce  the  department’s  costs  by  20% — or 
roughly  $20  billion  each  year  (Weigelt,  2012). 

The  complexity  of  traditional  DoD  acquisition  makes  the  process  difficult  for 
programs  with  tight  budgets  or  timelines  to  execute  predictably,  and  virtually  impossible  to 
execute  when  trying  to  meet  an  urgent  operational  need.  Given  this  situation,  how  can  the 
DoD  acquire  capabilities  both  faster  and  better?  The  answer  includes  expressing 
requirements  in  terms  of  general  capabilities  rather  than  firm  specifications  and  encouraging 
industry  to  respond  with  applicable  product  development  and  innovation  that  demonstrates 
best-of-breed  solutions. 

This  paper  suggests  Challenge-based  Acquisition  (ChBA)  as  an  approach  that  could 
be  applied  to  urgent  need  situations,  could  be  executed  in  a  more  rapid,  transparent 
manner,  and  would  allow  program  stakeholders  to  satisfy  mission  needs.  ChBA  presents 
challenges  to  a  set  of  interested  parties,  communicates  government  needs  to  the  private 
sector,  and  encourages  the  creation  of  innovative  products.  The  challenges  are  expressed 
in  terms  of  specific  capability  criteria  that  must  be  satisfied,  with  the  proposed  solutions 
proven  by  evidence  of  performance.  The  ChBA  approach  leverages  practices  designed  for  a 
rapidly  evolving  technology  environment  and  meets  the  real  demands  of  users  in  the  field.  It 
applies  acquisition  practices  and  techniques  necessary  to  achieve  better  outcomes  in  DoD 
programs  and  projects.  ChBA  is  founded  on  the  codification  of  government  needs 
expressed  as  concrete  performance  outcomes.  These  outcomes  are  challenges  that  are 
issued  to  a  marketplace  of  competing  vendors,  rather  than  needs  expressed  in  paper 
specification  documents  that  are  addressed  with  unproven  paper  proposals. 

Background 

End  users  have  difficulty  imagining  transformational  or  inventive  solutions  when  they 
have  a  working  solution  at  hand.  Soldiers,  for  example,  are  good  at  improvising  solutions  to 
address  shortcomings  of  equipment,  and  using  whatever  they  can  find  on  the  battlefield. 
Similarly,  they  are  experts  at  assessing  the  likely  success  of  incremental  improvements  to 
devices  and  techniques.  It  is  hard,  however,  to  extend  this  innovation  beyond  the  readily 
conceivable. 

Henry  Ford  supposedly  said,  “If  I  had  asked  people  what  they  wanted,  they  would 
have  said  ‘faster  horses’”  (Ford,  2006).  More  recently,  Steve  Jobs  said,  “You  can’t  just  ask 
customers  what  they  want  and  then  try  to  give  that  to  them.  By  the  time  you  get  it  built, 
they’ll  want  something  new”  (Burlingham  &  Gendron,  1989).  Even  the  brightest  equestrians 
would  have  had  trouble  picturing  the  utility  of  the  Model  T.  While  soldiers,  sailors,  and 
airmen  are  indeed  the  right  individuals  to  define  mission  requirements,  involving  them  in  the 
specification  process  can  limit  the  inventiveness  of  potential  solutions. 

But  suppose  that  Henry  Ford  had  heard,  “I  want  to  get  to  my  destination  faster  and 
with  comfort  and  affordability.”  In  this  case,  the  users  would  have  issued  a  concrete  mission 
challenge — get  there  faster  with  comfort — rather  than  a  specified  solution — a  faster  horse. 
Unfortunately,  government  acquisition  agents,  like  Ford’s  public,  rarely  think  in  terms  of 
mission  challenges  and  instead  think  in  terms  of  tighter  specifications  to  define  solutions. 

As  early  as  the  1980s,  the  DoD  recognized  that  relying  on  highly  rigid  specifications 
can  be  burdensome  and  costly.  Even  in  the  unusual  cases  where  specifications  and 
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standards  are  perfect,  premature  application,  over-application,  and  inappropriate  application 
of  standards  could  still  cause  complex  problems  (Bergman,  1996,  p.  32).  The  DoD  enacted 
acquisition  reforms,  deleting  many  military  specifications  from  contracts,  and  emphasizing 
outcome  and  performance-based  acquisitions  (Bergman,  1996). 

Challenges  present  an  option  for  achieving  these  goals.  Governments  and  industry 
have  long  used  challenges  to  spur  technology  advances  in  areas  that  include  agriculture, 
aviation,  energy,  medicine,  and  navigation.  For  example,  in  1714,  an  Act  of  Parliament 
established  the  British  Longitude  Prize  (Princeton  University,  n.d.).  The  Longitude  Board, 
which  administered  the  prize,  did  not  fund  technical  research  but  simply  promised  monetary 
awards  based  on  the  accuracy  of  proven  results:  £10,000  for  60  nautical  miles  of  accuracy, 
£15,000  for  40  nautical  miles,  and  £20,000  for  30  nautical  miles.  The  prize  prompted 
development  of  the  maritime  chronometer,  which  revolutionized  global  navigation  and 
solved  a  problem  that  had  bedeviled  seafaring  nations  for  over  150  years. 

The  Wright  Brothers’  contract  with  the  U.  S.  Army  (Smithsonian,  National  Air  and 
Space  Museum,  n.d.)  serves  as  a  20th  century  example  of  ChBA.  As  a  result  of  their 
airplane’s  performance  in  the  1909  U.S.  Army  flight  trials,  they  received  a  contract  that 
strongly  incentivized  speed,  with  a  10%  bonus  for  every  full  mile  per  hour  above  40.  The 
average  speed  of  the  Wrights’  aircraft  was  42.5  miles  per  hour,  earning  the  inventors  a 
$5,000  bonus  and  bringing  the  final  purchase  price  of  the  airplane  to  $30,000. 

For  decades,  the  aviation  industry  continued  to  create  ChBA-like  opportunities. 

When  aircraft  operators  abstracted  away  the  details  of  engine  design  and  simply  challenged 
power  plant  makers  to  deliver  performance  in  terms  of  thrust,  weight,  and  efficiency,  General 
Electric’s  Jack  Welch  conceived  the  idea  of  performance-based  logistics.  He  sold  “power  by 
the  hour”  (Knowledge@Wharton,  2007),  which  relieved  aircraft  owners  of  the  need  to 
inventory,  maintain,  and  repair  engines.  As  a  result,  the  costs  of  engine  inventories, 
maintenance,  and  repair  declined  dramatically. 

More  recently,  the  defense  and  aerospace  industries  have  used  challenges  to 
support  innovative  technology  development  in  areas  of  information  technology  (IT),  space 
transportation,  and  military  combat  systems,  as  illustrated  by  the  following  examples. 

Space  Transportation.  In  2004  Space  Ship  One,  a  suborbital  air-launched  space 
plane,  won  the  U.S.  $10  million  Ansari  X  Prize  by  completing  the  first  manned  private  space 
flight.  Space  Exploration  Technologies  Corporation,  also  known  as  SpaceX,  made  history  on 
May  25,  2012,  as  the  world's  first  privately  held  company  to  send  a  cargo  payload,  carried 
on  the  Dragon  spacecraft,  to  the  International  Space  Station  (SpaceX  Corporation,  n.d.). 

Military  Combat  Systems.  Mine  Resistant  Ambush  Protected  (MRAP)  vehicles  are 
a  family  of  armored  fighting  vehicles  originally  designed  under  the  guidance  of  the  U.S. 
Marine  Corps  to  survive  attacks  and  ambushes  involving  improvised  explosive  devices 
(lEDs).  On  July  31, 2007,  the  Marine  Corps  Systems  Command  launched  MRAP  II  pre¬ 
solicitation,  challenging  bidders  to  develop  a  new  vehicle  that  offered  a  higher  level  of 
protection  than  the  current  MRAP  vehicles.  The  U.S.  Army  Research  Laboratory  ensured 
the  technologies  used  in  the  Frag  Kit  6  (Fullerlove,  2009)  armor  upgrade  project  would  be 
available  to  MRAP  II  designers.  Initial  testing  at  the  Aberdeen  Proving  Grounds  disqualified 
vehicles  that  did  not  meet  requirements;  the  design  run-off  identified  two  vendors  whose 
vehicles  could  pass  the  demonstration  test. 

Information  Technology.  The  federal  and  commercial  markets  have  taken 
advantage  of  the  highly  competitive,  fast-paced  environment  of  IT.  Most  software 
manufacturers  must  prove  that  their  software  works  within  an  environment  and  that  it  can 
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integrate  into  a  larger  system.  Commercial  manufacturers  often  provide  free  demonstrations 
at  trade  shows  and  tabletop  exercises.  To  incorporate  vendor  solutions  into  its  Network 
Integration  Evaluation  (NIE)  program,  the  Army  conducts  semiannual  events  that  bring 
together  three  Army  communities  to  evaluate  militarily  useful  technologies  in  both  laboratory 
and  field  environments.  The  Army  applies  the  Agile  Process  to  accelerate  the  identification, 
testing,  and  fielding  of  relevant  networked  and  non-networked  capabilities  to  the  soldier,  in 
concert  with  capability  set  fielding  and  the  Army  Force  Generation  (ARFORGEN)  cycle. 

The  government  has  also  set  up  programs  specifically  designed  to  make  use  of 
challenges.  In  addition  to  the  Defense  Advanced  Research  Projects  Agency’s  well-known 
Grand  (DARPA,  2004)  and  Urban  (DARPA,  2008)  Challenges,  they  include  the  efforts 
summarized  in  the  following  section. 

Defense  Acquisition  Challenge  (DAC)  Program.  The  DAC  program  (Defense 
Acquisition  Challenge  [DAC]  Program,  2012,  §  2359b)  annually  solicits  technology 
proposals  from  small-  and  medium-sized  enterprises.  The  proposals  present  technologies 
that  could  lead  to  improvements  in  performance,  affordability,  manufacturing,  or  operational 
capability  if  introduced  into  existing  acquisition  programs  (DAC  Program,  2012,  §  2359b). 
The  new  technologies  should  replace  or  augment  some  aspect  of  a  current  procurement 
and  must  be  ready  off  the  shelf.  The  DAC  offers  a  promising  way  to  encourage  innovation 
and  help  new  companies  break  into  the  defense  market.  However,  it  centers  only  on 
improvements  to  existing,  conventional  acquisition  programs.  Ironically,  the  DAC  impels 
these  programs  to  expend  significant  resources  in  order  to  expose  opportunities  for 
innovation  that,  if  successful,  will  render  parts  of  the  original  acquisition  redundant.  In  a 
sense,  the  DAC  represents  an  example  of  ChBA  in  which  the  challenges  are  not  explicitly 
designed  by  the  government  but  inferred  by  industry  from  existing,  specification-based 
acquisitions.  However,  ChBA  has  the  advantage  of  permitting  entirely  fresh  approaches  and 
avoids  forcing  industry  to  accept  the  constraints  of  an  ongoing  acquisition. 

Defense  Innovation  Marketplace.  The  Defense  Innovation  Marketplace  serves  as  a 
centralized  resource  to  help  both  government  and  industry  “reinvigorate  innovation”  and 
fosters  collaboration  and  communication  between  government  and  industry  beyond 
traditional  Requests  for  Information  and  Industry  Days.  The  program  allows  industry  to  learn 
about  the  DoD’s  investment  priorities  and  capability  needs,  and  to  submit  summary  reports 
on  proprietary  Independent  Research  and  Development  (IR&D)  to  potential  customers.  For 
the  government,  the  Defense  Innovation  Marketplace  functions  as  a  one-stop  shop  for  DoD 
science  and  technology  planning,  acquisition  resources,  funding,  and  financial  information 
by  providing  agencies  with  search  tools  to  access  and  leverage  industry  technology  projects 
(DoD,  2013). 

Challenge.gov.  Outside  the  DoD,  the  Office  of  Management  and  Budget  (OMB)  has 
established  the  www.Challenge.Gov  website,  which  helps  individuals  and  companies  to 
compete  for  prizes  offered  by  various  government  agencies  for  solving  some  of  their 
toughest  problems.  The  website  supports  the  “OMB  Guidance  Memo  on  the  Use  of 
Challenges  and  Prizes  to  Promote  Open  Government,”  dated  March  2010.  That 
memorandum  responded  to  the  President’s  Directive  on  Transparency  and  Open 
Government,  which  tasked  the  OMB  Deputy  Director  for  Management  with  issuing  guidance 
for  the  increased  use  of  challenges  and  prizes  to  develop  new  tools  and  approaches  to 
improve  open  government.  OMB  launched  the  website  in  201 1  with  17  different  agencies 
posting  challenges  with  prizes,  including  a  recent  VA  $3  million  prize.  A  progress  report 
published  by  the  White  House  Office  of  Science  and  Technology  stated  that  prizes  may  be 
effective  in  stimulating  solutions  to  government  problems  (White  House  Office  of  Science 
and  Technology  Policy,  2012). 
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ChBA  Attributes  and  Benefits 

ChBA  creates  an  efficient  division  of  labor  where  the  government  focuses  on  what  it 
needs  (i.e.,  demand)  to  achieve  its  mission  and  private  industry  focuses  on  solutions  (i.e., 
supply).  The  government  could  use  ChBA  to  communicate  its  needs  by  framing  challenges 
that  are  analogous  or  identical  to  the  desired  capability.  Industry  could  then  respond  to  the 
challenges  without  being  confined  by  extraneous  constraints  such  as  highly  detailed 
engineering  specifications. 

To  meet  government  needs,  the  challenges  must  be  transparent  and 
understandable.  If  possible,  the  government  should  make  the  challenge  accessible  to  all 
parties  wishing  to  address  the  stated  needs.  Concrete  challenges  can  permit  nuanced  levels 
of  control  in  acquisition  not  possible  with  static  specifications  alone. 

As  shown  in  Table  1,  the  DoD  can  derive  several  benefits  from  applying  ChBA  in  its 
acquisitions.  They  include  expanding  user  involvement,  leveraging  technology,  reducing  risk 
through  proof  of  delivery  rather  than  paper-based  proposals,  accommodating  the  full  life 
cycle  of  a  fielded  system  or  product,  utilizing  the  most  appropriate  contracting  methods,  and 
engaging  industry  to  obtain  competitive  advantage. 
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Table  1.  Acquisition  Considerations,  ChBA  Compatibility,  and  Benefits 


Acquisition  Priority 

ChBA  Attribute 

ChBA  Benefits 

Urgent  Warfighter 
Mission  Needs  / 
Accelerated 

Fielding  Timeline 

ChBA  is  well  suited  to  meeting  urgent 
and  high-priority  requirements.  These 
needs  are  often  very  specific  and 
amenable  to  description  as  acquisition 
challenges.  Additionally,  the  urgency 
of  the  need  relaxes  most  of  the  DoD 
Instruction  5000.02  constraints.  (FAR 
6.302-2  Urgent  and  Compelling  Need). 

ChBA  allows  rapid  development  of 
advanced  technology,  including  both 
military  and  commercial  variants.  It  can 
result  in  fielding  the  correct  solution  the 
first  time,  and  avoiding  additional  costs  of 
rework  and  schedule  slippage — ideal  for 
meeting  urgent  warfighter  needs. 

Technological 

Maturity 

By  definition,  ChBA  requires  vendors 
to  offer  mature  technology  in  order  to 
participate  in  a  challenge  event. 

ChBA  allows  new  functionality  and 
interoperability  to  be  tested  in  a 
concurrent  environment,  ensuring  a  more 
operationally  ready  product  and  thus 
reducing  testing  costs  and  timelines. 

System  Life-Cycle 
Support  /  Upgrade 
Considerations 

ChBA  is  best  suited  for  technology¬ 
intensive  acquisitions,  which  are  likely 
to  be  short  lived  given  the  rapid  pace  of 
technology  evolution. 

ChBA  fits  well  into  short-duration 
programs,  where  constraints  in  the 
Operations  and  Support  phase  of  the 
Defense  Acquisition  Management  System 
process  become  irrelevant. 

Efficient 

Contracting 

Processes 

ChBA  can  be  executed  using  Broad 
Agency  Announcements  (BAAs), 
Indefinite  Delivery  /  Indefinite 

Quantity  (ID/IQ)  contracts,  Single 
Awards,  Blanket  Purchase  Agreements 
(BPAs),  or  Multi- Award  Contracts 
(MACs). 

ChBA  can  employ  a  flexible,  streamlined 
contracting  process  suited  to  a  variety  of 
contracting  vehicle  types.  This  enables  the 
program  manager  to  leverage  the 
contracting  type  that  best  suits  the 
program’s  needs  and  individual  tolerance 
for  risk. 

Enhanced  Industry 
Competition 

ChBA  is  structured  to  encourage  a 
diverse  range  of  industry  members 
(including  nontraditional  defense 
suppliers),  to  participate,  thus  making 
for  a  highly  competitive  environment. 

Because  ChBA  lowers  market  entry 
barriers  to  nontraditional  DoD  suppliers,  it 
provides  enhanced  opportunities  for 
competition  that  may  not  normally  arise 
within  the  traditional  defense  marketplace. 

Law,  Regulations,  Policy,  and  Guidance 

Recent  acquisition  laws,  regulations,  and  policies  emphasize  the  need  to  invest  in 
design  development  and  prototyping  to  mitigate  performance  risk  and  cost  growth  in  DoD 
acquisitions.  In  the  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of  2009  (Office  of 
the  Secretary  of  Defense,  2009),  Congress  directed  the  Secretary  of  Defense  to  ensure  that 
the  acquisition  strategy  for  each  major  defense  acquisition  program  includes  requirements 
to  demonstrate  capabilities  using  competitive  prototypes,  and  that  programs  consider 
appropriate  trade-offs  among  cost,  schedule,  and  performance  objectives  before 
development  begins. 

Likewise,  the  Federal  Acquisition  System  fully  supports  acquisition  challenges,  as 
indicated  by  the  guiding  principles  in  the  Federal  Acquisition  Regulations  (FAR  1 .102). 
Specifically,  federal  acquisitions  must  satisfy  customer  needs  in  terms  of  cost,  quality,  and 
timeliness  of  the  delivered  product  or  service  by 
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•  maximizing  the  use  of  commercial  products  and  services; 

•  using  contractors  who  have  a  track  record  of  successful  past  performance  or 
who  demonstrate  a  current  superior  ability  to  perform; 

•  promoting  competition; 

•  minimizing  administrative  operating  costs; 

•  conducting  business  with  integrity,  fairness,  and  openness;  and 

•  fulfilling  public  policy  objectives. 

FAR  Part  2.101,  Definitions,  includes  the  following  provision:  “Qualification 
requirement  means  a  Government  requirement  for  testing  or  other  quality  assurance 
demonstration  that  must  be  completed  before  award  of  a  contract.”  The  FAR  and  the 
Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  contain  regulatory  and  policy 
guidance  to  allow  testing  of  designs  before  implementation  and  fielding.  FAR  1 1 .801 ,  Pre¬ 
award  in-use  evaluation,  states  that  “supplies  may  be  evaluated  under  comparable  in-use 
conditions  without  a  further  test  plan,  provided  offerors  are  so  advised  in  the  solicitation.  The 
results  of  such  tests  or  demonstrations  may  be  used  to  rate  the  proposal,  to  determine 
technical  acceptability,  or  otherwise  to  evaluate  the  proposal.” 

DoD  Directive  5000.01  (DoD,  2003a)  requires  each  military  department  to  establish 
its  own  independent  Operational  Test  Agency  (OTA)  to  plan  and  conduct  operational  tests, 
report  results,  and  provide  evaluations  for  effectiveness  and  suitability.  DoDD  5000.01 
(DoD,  2003a)  further  requires  the  integration  of  test  and  evaluation  throughout  the  defense 
acquisition  process.  DoD  Instruction  5000.02  (DoD,  2008),  issued  in  2008,  requires  a 
Materiel  Development  Decision  prior  to  a  program’s  entry  into  the  acquisition  process, 
causing  program  offices  to  invest  more  funds  to  mitigate  technical  risk.  Such  requirements 
support  the  use  of  ChBA  as  a  means  to  improve  testing  efficiency  and  effectiveness  across 
DoD  OTAs  (DoD,  2003a). 

The  examples  described  previously  show  that  acquisition  law  and  regulation  already 
allow  demonstration  testing  to  ensure  contractor  performance.  Precedents  in  which  the 
government  has  successfully  applied  ChBA  techniques  to  acquisitions  exist  in  several 
domains,  such  as  IT  and  space.  Thus,  applying  ChBA-like  methods  to  satisfy  critical  needs 
appears  both  legal  and  practical. 

An  initial  review  of  acquisition  regulation  and  policy  reveals  when  and  how  ChBA 
may  be  best  applied. 

•  Research  and  development:  A  Broad  Agency  Announcement  (BAA) 
procedure  provides  a  competitive  acquisition  process.  If  the  challenge 
involves  seeking  innovative  solutions,  then  it  almost  certainly  falls  within  the 
area  of  early  exploration  or  development. 

•  Components,  sub-systems,  or  items:  The  smaller  an  acquisition,  the  easier  it 
is  to  adapt  to  the  acquisition  process  without  the  multi-layered  FAR  (2013)  or 
DoD  Instruction  5000.02  (DoD,  2008)  provisions  or  constraints. 

•  Urgent  capability:  Field  commanders  who  require  rapid  action  express  their 
urgent  wartime  needs  in  Joint  Urgent  Operational  Needs  Statements  or 
similar  documents.  These  needs  are  often  very  specific  and  amenable  to 
description  as  acquisition  challenges. 
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•  Short  life  cycle:  Technology-intensive  acquisitions  are  likely  to  be  short  lived 
given  the  rapid  pace  of  technology  evolution.  This  makes  the  complex 
guidance  regarding  the  importance  of  reducing  long  life-cycle  costs  during  the 
Operations  and  Support  phase  of  the  Defense  Acquisition  Management 
process  essentially  irrelevant. 

Better  Buying  Power  2.0 

Recent  DoD  guidance  has  also  emphasized  a  faster  approach  to  adopting  solutions 
by  using  rapid  acquisition  or  agile  techniques.  In  his  “Better  Buying  Power”  memorandum 
(USD[AT&L],  2010),  the  Undersecretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  recognized  the  need  to  make  DoD  acquisitions  more  affordable  through  added 
investment  at  the  beginning  of  the  acquisition  process  to  ensure  a  cost-competitive  result. 
The  Defense  Better  Buying  Power  (BBP)  2.0  initiative  (USD[AT&L],  2012)  covers  several 
areas  in  which  challenges  can  be  well  suited  to  implement  current  guidance. 

Achieve  Affordable  Programs 

Mandate  Affordability.  Challenges  can  be  used  to  mandate  affordability  by 
requiring  that  all  solutions  meet  a  specific  price  target  as  a  condition  of  participation  in  the 
challenge  and  subsequent  procurement.  For  example,  a  challenge  may  specify  that  the 
chosen  solution  shall  not  cost  more  than  X  dollars.  Challenge  participants  may  automatically 
become  ineligible  for  a  final  contract  award  unless  their  solutions  meet  the  unit  cost  and/or 
total  cost  requirements.  This  approach  ensures  that  all  solutions  that  the  government 
procures  using  ChBA  will  meet  pre-defined  program  affordability  targets. 

Reduce  Program  Cost  and  Risk.  The  government  can  use  challenges  to  reduce 
risk  through  “actual”  demonstrated  performance  before  the  government  commits  itself  to  a 
long-term  contract.  Furthermore,  the  DoD  can  build  testing  and  certification  criteria  into  the 
challenge  event,  thereby  ensuring  that  accepted  solutions  will  meet  testing  requirements 
and  required  performance  objectives  before  they  are  purchased  by  the  government,  thus 
reducing  risk,  delivery  timelines,  and  cost. 

Incentivize  Productivity  and  Innovation  in  Industry 

Incorporate  Innovation  Into  Production  at  a  More  Rapid  Rate.  Challenges  can 
spur  industry  productivity  by  guiding  efficient  application  of  research  and  development 
resources  to  meet  specific  requirements  for  a  concrete  capability.  Furthermore,  because  the 
technology  purchased  must  be  nearly  production  ready  at  the  time  the  challenge  takes 
place,  this  mechanism  provides  an  additional  incentive  for  industry  to  establish  an  efficient 
production  process  that  drives  down  costs  and  promotes  efficiency. 

Promote  Effective  Competition 

Emphasize  Competition  Strategies  and  Create/Maintain  Competitive 
Environments.  ChBA  directly  supports  creation  of  a  competitive  acquisition  environment 
because  it  encourages  a  wide  range  of  solution  providers  to  participate.  Challenges  must  be 
open  to  the  greatest  possible  number  of  potential  participants,  since  traditional  requirements 
for  entering  the  defense  market  do  not  apply  in  the  ChBA  environment.  For  example,  in  a 
challenge  focused  on  current  performance  requirements,  previous  experience  may  be 
irrelevant  when  it  comes  time  to  make  a  contract  award.  This  key  difference  enables 
organizations  and  even  individuals  who  have  little/no  defense  experience  to  participate,  thus 
enlarging  the  competitive  landscape. 

Enforce  Open  Systems  Architectures  and  Manage  Technical  Data  Rights.  The 

DoD  can  also  use  challenges  effectively  to  support  the  introduction  of  open  system 
architectures  (OSAs)  across  the  DoD.  OSAs  require  a  predefined  architecture  with  open 
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interfaces  for  easy  integration  of  components  (DoD,  2011).  Challenges  can  be  used  to 
develop  adaptable  technology  for  key  components  of  open  systems.  ChBA  also  permits 
flexible  intellectual  property  arrangements  and  opportunities  for  licensing  negotiations  that 
support  effective  management  of  technical  data  rights  over  the  program  life  cycle. 

Roles  and  Responsibilities 

Government  Role 

The  government  takes  on  a  new  role  in  ChBA.  In  traditional  acquisition,  the 
government  communicates  its  needs  in  a  specification  and  must  assume  that  fulfillment  of 
the  specification  equates  to  meeting  mission  needs.  However,  the  specification  could  be 
appropriately  constrained,  under  constrained,  over  constrained,  or  simply  wrong.  If  the 
specification  is  under  constrained  or  wrong,  the  result  is  unlikely  to  meet  mission 
requirements.  If  the  specification  is  over  constrained,  the  solution  will  likely  not  be  optimal 
and  might  be  impossible  to  implement. 

Current  incentives  encourage  contractors  to  propose  solutions  to  meet  over¬ 
constrained  specifications,  even  if  the  constraints  create  a  high  risk  of  failure  and,  in  the 
process,  spend  large  amounts  of  money  on  developing  solutions  that  may  never  be  fully 
realized.  The  fundamental  flaw  in  this  process  is  the  failure  to  recognize  when  over- 
specifying  drives  design.  To  avoid  these  problems  and  implement  ChBA  successfully,  the 
government  should  consider  the  following: 

Decompose  Complex  Requirements  Into  Challenges.  The  government  will  need 
to  interpret  warfighter  requirements  and  translate  them  into  meaningful  challenge  events 
that  will  give  industry  the  latitude  for  innovation  and  get  users  what  they  need.  This  requires 
the  government  to  have  a  broad  vision  and  a  commitment  to  success  beyond  that  typically 
needed  to  issue  requests  for  proposals  or  BAAs.  Furthermore,  the  government  should 
ensure  that  technical  details  are  not  over  specified,  but  rather  generalized  into  technology- 
agnostic  capability  requirements  that  can  be  demonstrated  in  a  challenge. 

Generalize  User  Experience  and  Needs  and  Communicate  Them  to  Industry. 

After  gathering  requirements  from  the  warfighter  and  translating  them  into  executable 
challenges,  the  government  should  communicate  the  scope  of  the  challenges  to  industry.  In 
doing  so,  the  government  admittedly  assumes  risk,  because  formulating  the  challenges 
requires  the  ability  to  interpret  and  translate  warfighter  experience  and  needs  in  a  clear  and 
concise  manner,  thus  enabling  industry  to  execute  the  challenge. 

Find  Unclassified  Analogues  to  Classified  Situations.  The  government  should 
employ  ChBA  to  identify  possible  solutions  to  classified  requirements  by  utilizing 
unclassified  challenge  analogues.  In  these  situations,  participants  may  not  know  the  details 
of  the  particular  setting  in  which  the  government  plans  to  use  a  solution,  and  instead  would 
only  know  the  general  performance  objectives  to  be  met.  This  approach  supports  an 
enhanced  competitive  environment  by  enabling  those  vendors  that  do  not  possess  the 
required  security  clearances  and  facilities  to  participate  in  the  challenge. 

Design  and  Execute  a  Concrete  Challenge  Apparatus.  The  government  should 
design  challenge-specific  execution  and  evaluation  processes  that  include  a  plan  for 
communicating  challenges  to  industry,  a  plan  detailing  how  the  challenge  will  be  executed 
contractually,  specific  requirements  for  challenge  participation,  and  detailed  evaluation 
criteria  to  ensure  the  challenge  evaluation  will  be  fair  to  all  participants. 

Perform  Quantitative  and  Qualitative  Analysis  of  Challenge  Results.  The 

government  should  use  quantitative  and  qualitative  measurements  to  evaluate  challenge 
results.  More  specifically,  the  government  may  evaluate  the  challenge  participants  during  or 
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immediately  after  the  challenge,  and/or  over  a  longer  term,  as  defined  by  the  initial 
challenge  notice.  Upon  completion  of  the  challenge,  the  government  may  opt  to 

•  Purchase  one  or  more  of  the  competitors’  offerings  based  on  confidence  in 
the  product’s  utility,  as  demonstrated  during  the  challenge. 

•  Refine  and  reissue  the  challenge  based  on  lessons  learned  during  challenge 
performance.  This  can  become  part  of  an  incremental  government  strategy 
that  includes  challenge-based  research  projects. 

•  Do  nothing.  If  the  challenge  results  did  not  inspire  confidence  that  any  of  the 
products  would  meet  government  needs,  the  government  has  no  obligation  to 
let  a  contract.  This  prevents  a  potentially  unsuccessful  acquisition. 

Industry  Role 

Industry  also  takes  on  a  new  role  in  ChBA:  one  that  more  closely  mirrors  how 
industry  normally  develops  and  brings  a  product  to  the  commercial  market  versus  the 
traditional  defense  acquisition  market.  In  this  case,  industry  would  be  responsible  for 
independently  developing  a  solution  that  addresses  a  given  capability  need  (e.g.,  “get  to  my 
destination  faster  and  with  comfort  and  affordability”).  This  approach  contrasts  starkly  with 
the  traditional  defense  acquisition  process  whereby  the  government  provides  detailed 
specifications  and  requirements  (e.g.,  faster  horses)  to  industry.  In  the  former  case,  industry 
bears  most  of  the  risk,  while  in  the  latter  case  the  risk  is  borne  by  the  government.  Thus,  in 
support  of  ChBA,  industry  should  do  the  following: 

Innovate.  ChBA  will  demand  that  industry  propose  innovative  solutions.  ChBA  is  by 
definition  technology  agnostic — it  does  not  presuppose  one  specific,  ideal  technological 
solution.  Consequently,  government  will  not  prescribe  a  specific  technological  path  that 
industry  must  follow,  but  rather  will  present  its  requirements  in  the  form  of  general  challenge 
objectives  that  must  be  met.  Industry  must  then  apply  its  expertise  to  determine  the  best 
technical  approach  to  address  the  objectives  within  the  schedule/cost  constraints  provided 
by  the  government. 

Cooperate  With  Traditional/Non-Traditional  Entities.  No  single  company  has  a 
monopoly  on  innovative  solutions.  ChBA  acquisition,  by  its  very  definition,  seeks  the  best 
technology  to  address  the  military’s  toughest  problems.  Therefore,  industry  must  be  willing 
to  cooperate  with  any  individual  or  organization  that  could  contribute  to  a  solution  meeting 
challenge  performance  criteria. 

Dedicate  R&D  Funding.  ChBA  will  require  that  industry  dedicate  IR&D  funding  to 
develop  a  solution  that  meets  challenge  performance  criteria.  While  the  government  may 
choose  to  provide  nominal  funding  to  enable  organizations  to  attend  and  participate  in 
challenge  events,  it  may  not  necessarily  fund  any  of  the  initial  development  effort. 

Negotiate  Intellectual  Property  Licenses.  ChBA  will  require  that  industry  be 
prepared  to  negotiate  potential  intellectual  property  licenses  with  the  government.  As  a 
result,  it  is  important  that  industry  properly  identify  which  of  its  solutions  it  derived  through 
exclusive  use  of  IR&D  funding  versus  those  that  may  have  been  developed  at  partial  or  full 
government  expense.  Such  a  distinction  is  important,  because  the  source  of  funding  dictates 
the  type  of  licensing  rights  available  to  the  government. 

ChBA  Within  Defense  Acquisition 

ChBA  is  well  suited  to  smaller  acquisitions,  which  are  usually  not  controlled  by  the 
full  DoD  Instruction  5000.02  (DoD,  2008)  guidance.  In  large  acquisitions,  ChBA  can 
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enhance  the  standard  process  by  efficiently  providing  many  of  the  5000.02-specified 
components,  if  not  necessarily  the  entire  solution. 


Since  ChBA  is  grounded  in  requirements  development  and  the  acquisition  process,  it 
does  not  represent  a  radical  or  disruptive  break  with  accepted  practice.  Instead,  it 
generalizes  and  builds  on  existing  concepts  such  as  the  Defense  Acquisition  Management 
System  (DAMS),  which  guides  the  procurement  of  major  military  systems.  Figure  1  provides 
a  graphical  view  of  the  DAMS  phases. 
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Figure  1.  DAMS  Phases 

The  DAMS  recognizes  the  need  for  an  evolutionary  approach  to  acquisition,  stating 
that  “an  evolutionary  approach  delivers  capability  in  increments,  recognizing,  up  front,  the 
need  for  future  capability  improvements”  (DAU,  2011).  Increments  are  managed  through 
repeated  application  of  the  Technology  Development  and  Engineering  and  Manufacturing 
Development  phases.  ChBA  applies  in  these  early  phases  of  the  DAMS  and  in  the  general 
evolutionary  approach.  Specific  opportunities  for  ChBA  application  within  the  DAMS  are 
further  described  in  Table  2. 
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Table  2. 


DAMS  and  ChBA 


DAMS  Phase 

Applicability  of  ChBA 

Materiel  Solution  Analysis — Assess  potential 
materiel  solutions  and  perform  an  Analysis  of 
Alternatives.  This  phase  begins  when  an  Initial 
Capabilities  Document  is  approved  that  contains 
an  analysis  of  current  mission  performance  and 
potential  concepts  from  across  the  DoD.  It  ends 
when  the  Analysis  of  Alternatives  is  complete  and 
materiel  solution  options,  identified  in  the  Initial 
Capabilities  Document,  are  recommended. 

The  Analysis  of  Alternatives  enumerates  the  critical 
elements  needed  by  each  proposed  materiel  solution. 
ChBA  supplements  this  step  because  industry  provides 
the  technology  needed  to  create  a  capability  prior  to 
participation  in  the  challenge.  If  the  government  does 
become  involved  in  selecting  and  maturing  technologies, 
a  challenge,  based  on  the  needed  capability,  could  be 
used  to  explore  the  range  of  candidate  technologies  and 
assess  their  maturity. 

Technology  Development — Determine  and  mature 
the  appropriate  technologies  needed  for  the  full 
system.  Critical  technology  elements,  identified  in 
the  previous  phase,  must  be  demonstrated  using 
prototypes.  The  Technology  Development  phase 
requires  the  creation  of  a  Technology 

Development  Strategy.  For  an  evolutionary 
acquisition,  the  Technology  Development  Strategy 
is  to  include  a  preliminary  description  of  how  the 
materiel  solution  will  be  divided  into  acquisition 
increments  based  on  mature  technology  and  an 
appropriate  limitation  on  the  number  of  prototype 
units. 

A  ChBA  approach  to  the  Technology  Development 
Strategy  is  to  design  a  challenge  that  proves  the  maturity 
of  each  needed  technology.  The  challenge  may  or  may 
not  require  a  prototype,  but  will  place  emphasis  on 
attainment  of  the  technological  capability  rather  than  the 
delivery  of  a  prototype.  The  acquisition  increment 
requirement  of  the  Technology  Development  Strategy 
can  be  served  by  a  standing  challenge  that  persists 
through  time  as  multiple  challengers  demonstrate  a 
range  of  solutions.  A  standing  challenge  gives  industry  a 
chance  to  improve  on  existing  solutions.  It  also 
encourages  the  discovery  of  game-changing  solutions  to 
challenges  that  have  already  been  solved  with  more 
pedestrian  technologies. 

Engineering  and  Manufacturing  Development — 
Develop  the  full  system  or  some  increment  of  the 
full  system  capability.  This  includes  full  system 
integration  and  creation  of  an  affordable  and 
executable  manufacturing  process. 

ChBA  potentially  eliminates  the  need  for  this  phase 
because  the  technology  needed  to  create  a  capability  is 
already  at  or  near  full  capability  as  a  prerequisite  for 
challenge  participation.  Further,  the  challenge  may 
specifically  require  that  participants  (or  their  partners) 
produce  fully  operational  versions  of  the  submissions  by 
a  certain  point  in  time  following  the  challenge  event. 

Production  and  Deployment — Achieve  an 
operational  capability  that  satisfies  mission  needs. 
This  includes  low  rate  production  for  evaluation  of 
major  systems  and  full  production  or 
procurement  of  smaller  systems. 

Technology  acquired  using  ChBA  is  by  definition  nearly 
production  ready;  therefore,  ChBA  can  be  used  to 
accelerate  the  LRIP  portion  of  the  acquisition  process. 
Furthermore,  if  operational  testing  and  evaluation 
criteria  are  already  built  into  the  challenge  construct, 
technology  will  have  met  T&E  requirements  before  the 
government  makes  a  buy  decision — again  accelerating 
the  IOT&E  part  of  the  acquisition  process. 

Operations  and  Support — Execute  a  support 
program  that  meets  readiness  and  operational 
requirements  and  sustains  the  system,  in  a  cost- 
effective  manner,  over  its  total  life  cycle.  This 
phase  also  includes  disposal  of  the  system  at  the 
end  of  its  life. 

Challenges  can  be  designed  to  ensure  that  operations  and 
support  requirements  are  built  in  from  the  beginning.  As 
such,  a  challenge-based  demonstration  can  reenact 
operational  requirements  for  readiness  and  sustainment 
to  demonstrate  capability  before  the  government  makes 
a  commitment  to  purchase. 

The  ChBA  Process 

Figure  2  shows  the  flow  of  a  hypothetical  challenge-based  acquisition.  The  process 
begins  when  the  government  becomes  aware  of  a  user’s  need.  The  acquiring  agency,  or  its 
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technical  support  organization,  postulates  a  capability  that  can  satisfy  the  user’s  need.  This 
is  a  creative  process  and  requires  more  technical  insight  than  simply  recording  what  the 
user  has  requested. 

With  a  desired  capability  in  mind,  the  government  agency  creates  a  set  of  concrete 
performance  challenges  that  would  demonstrate  the  ability  of  the  envisioned  capability  to 
solve  the  user’s  problem.  For  example,  the  user  problem  could  be  that  soldiers  need  better 
situational  awareness  when  fighting  in  urban  areas.  The  envisioned  capability  could  be  an 
information  sharing  mechanism.  A  supporting  challenge  might  be  to  show  that  solders  who 
use  the  candidate  challenge  solution  earn  better  scores  in  urban  combat  training  than  those 
who  do  not  use  the  solution. 
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Figure  2.  ChBA  Process 

The  challenge  event  can  range  from  large,  periodic,  public  occasions  to  private,  one¬ 
time  visits  to  a  testing  laboratory. 

At  Arrow  C  in  Figure  2,  industry  decides  to  attempt  the  challenge.  This  may  produce 
two  results: 


•  Increased  government  knowledge  of  potential  solutions  and  their  vendors, 
depicted  by  Arrow  D. 

•  Greater  understanding  of  the  trade  space  in  which  a  solution  might  be  found. 
Arrows  E  and  F  show  that  this  understanding  comes  from  both  observed 
performance  in  the  challenge  event  and  information  available  about 
promising  vendors  and  their  products. 

Arrows  G  and  H  show  that  ChBA  can  be  a  cyclic  process. 

•  Competitors  whose  product  failed  in  one  challenge  may  make  another 
attempt  after  modifying  their  products.  The  government  may  also  take  this 
opportunity  to  fund  promising  vendors  directly.  Direct  funding  rewards 
vendors  for  their  initiative  and  incentivizes  them  to  attempt  the  challenge 
again,  as  depicted  by  Arrow  G. 

•  Based  on  improved  knowledge  of  the  needed  capability  and  the  technical 
trade  space,  the  government  can  revise  the  challenge  and  begin  the  process 
again,  as  depicted  by  Arrow  H.  This  can  be  important  during  the  acquisition  of 
complex  systems,  where  multiple  steps  may  be  needed  to  state  the  challenge 
correctly  or  arrive  at  the  appropriate  technology. 
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Case  Study — Joint  IED  Defeat  Organization 

The  mission  of  the  Joint  IED  Defeat  Organization  (JIEDDO,  n.d.-b)  is  to  “reduce  the 
effectiveness  and  lethality  of  lEDs,  to  allow  freedom  of  maneuver  for  joint  forces,  federal 
agencies,  and  partner  nations  in  current  and  future  operating  environments”  (JIEDDO,  n.d.- 
a).  In  its  strategic  plan,  JIEDDO  identifies  as  one  of  its  enduring  capabilities  the  ability  to 
“employ  authorities,  flexible  resources,  streamlined  processes,  and  effective  oversight  to 
drive  the  research  and  development  community  to  rapidly  field  C-IED  solutions”  (JIEDDO, 
n.d.-a).  The  computer  screen  saver  depicted  in  Figure  3  carries  JIEDDO’s  fundamental 
message  to  the  staff  every  day.  This  intensity  of  purpose  and  need  for  rapid  action  make 
JIEDDO  well  suited  to  apply  ChBA. 


Figure  3.  JIEDDO  Organization-Wide  Computer  Screen  Saver 

In  the  summer  of  201 1 ,  JIEDDO  faced  the  sudden  need  for  a  particular  class  of  robot 
in  the  war  in  Afghanistan.  JIEDDO  demonstrated  strength  and  resolve  by  issuing  concrete 
challenges  that  communicated  the  soldiers’  needs  rather  than  reading  vendor  literature  and 
attending  presentations.  The  challenges  were  drawn  from  the  suite  of  Response  Robot 
Performance  Standards  (National  Institute  of  Standards  and  Technology,  2011)  developed 
by  the  National  Institute  of  Standards  and  Technology  (NIST;  www.nist.gov).  The  NIST  test 
method  suite  includes  a  range  of  mobility  and  duration  assessment  devices  that  provide 
excellent  models  of  the  challenges  robots  face  in  Afghanistan. 

Six  vendors  accepted  the  challenge  and  at  their  own  expense  brought  their  robots  to 
NIST  for  assessment.  Some  robots  met  the  challenges  as  their  vendors  claimed.  Other 
systems  displayed  large  gaps  between  promised  capability  and  demonstrated  performance. 
JIEDDO  then  presented  the  results  of  the  challenges  and  the  concrete  characteristics  of  the 
robots  to  field  users  in  Afghanistan. 

JIEDDO  discovered  that  the  original  request  from  the  field  had  been  over 
constrained.  The  challenge  performance  helped  the  users  to  understand  the  performance 
trade  space  and  to  recognize  that  one  class  of  robot  alone  would  not  meet  their  needs.  As  a 
result,  JIEDDO  identified  two  classes  of  robot  that  addressed  the  concerns  of  two  distinct 
user  communities — an  important  distinction  nowhere  to  be  found  in  the  original  field  request. 

In  addition  to  clarifying  what  the  users  really  needed,  the  challenge  process 
encouraged  vendors  to  improve  their  products  before  the  government  committed  itself  to  a 
purchase.  The  challenge  brought  transparency  and  mutual  vendor  visibility,  sparking 
beneficial  competition  and  product  improvement.  Within  months,  vendors  asked  to  return  to 
the  NIST,  again  at  their  own  expense,  for  another  opportunity  to  confront  the  challenges  and 
improve  JIEDDO’s  perception  of  their  products’  quality.  In  this  way,  ChBA  enabled  JIEDDO 
to  go  from  the  initial  request  for  help  to  fielded  systems  in  less  than  a  year. 
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Implementation 

Barriers 

Barriers  to  implementation  are  rooted  in  the  possibility  that  the  government  will 
attempt  to  manage  ChBA  in  the  same  way  it  manages  a  traditional  technology  acquisition. 
While  ChBA  leverages  the  DAMS  and  supporting  processes,  the  acquisition  pitfalls  that 
plague  these  traditional  systems  could  equally  undermine  ChBA. 


Table  3.  Acquisition  Attributes  and  Implications  for  ChBA 


Typical  Acquisition  Pitfalls 

ChBA  Implications 

Mission  needs  are  incorrectly  translated  through  the 
daisy  chain  of  performance-related  documentation, 
resulting  in  wrongly  defined  system  performance  that 
is  over  specified,  driving  non-optimal  solutions. 

The  DoD  may  not  be  able  to  acquire  the  most 
innovative  solutions  from  industry  using  ChBA  if  the 
government  dictates  specific  requirements  instead  of 
describing  generic  capabilities  to  be  demonstrated  at  a 
challenge  event. 

The  competitive  nature  of  funding  motivates  the 
government  to  make  optimistic  predictions  of  system 
performance  in  order  to  obtain  program  approval. 
Likewise,  industry  is  incentivized  to  propose  risky 
solutions,  since  this  can  lead  to  long-term  lock-in  and 
opportunities  for  contract  modifications  to  address 
product  shortcomings. 

ChBA  fundamentally  does  not  permit  either  the 
government  or  industry  to  over-promise  system 
performance.  Performance  must  be  proven  in  a 
transparent  manner  prior  to  the  buy  decision. 

The  government  often  approaches  acquisition  in  a  risk- 
averse  manner,  requiring  extended  periods  of  risk 
reduction  accompanied  by  documentation  requiring 
multiple  reviews.  Regardless  of  risk-reduction  efforts, 
real  risk  to  the  government  buyer  exists  due  to  the  late 
conduct  of  the  Operational  Evaluation. 

ChBA  addresses  these  risks  up  front  and  is  ideally 
suited  to  high-risk  technological  solutions.  However, 
ChBA  will  require  cooperation  from  current  document 
owners  and  the  Operational  Test  community  to  avoid 
this  pitfall. 

Adopting  ChBA 

In  order  for  the  DoD  to  adopt  and  universally  implement  ChBA  across  the  broader 
defense  enterprise,  we  recommend  that  the  DoD  do  the  following: 

•  Educate  acquisition  professionals  about  ChBA.  There  is  a  gap  between 
the  latitude  allowed  by  current  acquisition  law  and  the  state  of  acquisition 
practice.  Briefly  stated,  the  defense  acquisition  community  culture  tends  to  be 
highly  risk  averse  even  when  there  are  logical  arguments  to  take  on 
additional  risk.  This  cultural  dynamic  is  reinforced  as  program  managers 
regularly  spend  money  to  reduce  uncertainty  (e.g.,  risk;  Frick,  2010,  p.  364). 
ChBA  enables  the  government  to  explore  potentially  high-risk/high-reward 
solutions  in  a  low-risk  environment  before  vast  resources  are  dedicated  to  an 
acquisition  effort.  This  suggests  that  the  acquisition  corps  needs  to  be 
educated  on  the  value  of  using  ChBA  in  these  circumstances. 

•  Publicize  examples  of  ChBA  success.  The  government  should  publicize 
working  examples  of  ChBA  within  the  acquisition  and  supporting  professional 
communities.  Acquisition  professionals  will  feel  more  comfortable  embracing 
ChBA  if  they  can  point  to  other  successful  programs  that  use  ChBA 
strategies.  Senior  leadership  must  be  convinced  of  ChBA  utility  so  that  they 
will  commission  a  few  early  adopter  programs,  and  the  managers  of  these 
early  adopter  programs  must  operate  under  senior  leadership  imprimatur  and 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-217- 


protection.  The  success  of  the  early  adopters  will  then  encourage  more 
cautious  program  managers  to  follow  suit,  provided  the  results  of  ChBA  are 
widely  publicized  across  the  DoD. 

•  Develop  a  ChBA  desk  guide  as  a  reference  for  acquisition 
professionals.  The  DoD  should  produce  a  ChBA  desk  guide  to  support  use 
of  ChBA  across  the  defense  enterprise.  The  guide  should  be  patterned  after 
existing  acquisition  desk  guides  to  answer  day-to-day  questions  and  provide 
example  solutions,  practices,  and  business  cases  related  to  ChBA.  As  ChBA 
is  more  widely  adopted  across  the  DoD,  the  desk  guide  should  be  updated 
periodically  to  document  new  lessons  learned,  case  studies,  and  best 
practices. 

•  Consider  legislative  and  regulatory  change.  Amend  the  FAR  and  revise 
current  acquisition  guidance  to  reflect  ChBA  as  an  accepted  method  to 
acquire  capability  for  the  warfighter.  Explicit  acceptance  of  ChBA  in  published 
regulatory  and  policy  documents  will  codify  the  approach  and  bring 
recognition  that  it  represents  a  sound  way  of  doing  business  and  can  achieve 
high  impact  in  performance  improvement. 

Conclusion 

ChBA  can  solve  a  class  of  acquisition  problems  defined  by  industry’s  tolerance  of 
capital  risk  and  the  government’s  ability  to  express  user  needs  in  terms  of  concrete 
challenges.  It  thus  constitutes  a  logical  next  step  in  the  current  wave  of  acquisition  reforms. 
ChBA  has  proven  itself  in  the  world  of  civilian  advanced  technology  acquisition  and  has 
been  demonstrated  successfully  in  limited  areas  within  the  DoD. 

Successful  application  of  ChBA  demands  a  renewed  government  commitment  to 
technical  involvement  in  acquisition,  calling  upon  the  acquisition  agent  to  create  challenges 
that,  if  fulfilled,  would  also  meet  the  user’s  requirements.  This  requires  a  clear  understanding 
of  user  need,  as  well  as  the  creativity,  imagination,  and  technical  insight  necessary  to  design 
the  challenge. 

ChBA  encourages  the  best  performance  in  industry  by  freeing  companies  from 
constraints  unrelated  to  challenge  success.  It  encourages  new  players  to  participate  and 
creates  a  level  playing  field  for  all  involved.  ChBA  adheres  to  government  regulations  and  is 
practical  to  use  within  the  current  federal  acquisition  system.  Above  all,  it  offers  an  efficient 
means  for  stimulating  industrial  innovation  and  reducing  the  time  and  cost  of  government 
acquisition  programs. 
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Abstract 

This  report  describes  the  preliminary  analysis  and  findings  of  our  study  exploring  what  drives 
successful  organizational  adaptation  in  the  context  of  technology  transition  and  acquisition 
within  the  Department  of  Defense  (DoD).  It  is  based  on  our  initial  collection  and  analysis  of 
archival  and  interview  data.  We  began  this  study  seeking  to  understand  what  influences  the 
successful  transition  of  commercial  off-the-shelf  (COTS)  technologies  to  the  warfighter, 
focusing  on  the  Joint  Capabilities  Technology  Demonstration  (JCTD)  office  as  a  successful 
case  study.  In  the  course  of  our  investigation,  we  noted  shifts  in  organization  structure,  goals, 
and  business  processes  of  the  JCTD  in  response  to  changing  needs  of  warfighters  in  Iraq 
and  Afghanistan.  Further  exploration  indicated  that  these  shift  were  not  unique  to  the  JCTD, 
but  were  one  example  of  many  adaptive  solutions  to  changing  needs  faced  by  the  DoD 
acquisition  community.  This  led  us  to  focus  our  research  on  better  understanding  what  drives 
successful  organizational  adaptation.  Our  preliminary  analysis  suggests  that  ad  hoc  problem 
solving  may  be  an  undervalued  yet  broadly  practiced  skill  set  within  the  DoD,  which  may 
support  adaptive  responses  to  change  by  the  acquisition  community.  We  are  currently 
collecting  additional  data,  which  we  will  use  to  further  explicate  our  findings. 

Introduction 

Defense  acquisition  is  a  key  technical  and  business  function,  vital  to  the  success  of 
the  U.S.  military.  However,  it  is  also  the  focus  of  seemingly  constant  critique  and  reform. 
Most  recently,  the  rapidly  changing  global  environment  and  tactics  of  adversaries  have 
highlighted  gaps  in  the  organization’s  business  process  capability,  intensifying  the  calls  for 
process  reform.  It  is  widely  recognized  that  DoD  acquisition  must  become  more  nimble  and 
flexible  to  more  rapidly  deploy  materiel  solutions  to  new  and  emerging  problems  and  that 
doing  so  will  require  changes  in  organization  structure,  culture,  and  processes.  What  is  less 
clear  is  how  to  gain  the  most  value  from  investment  in  change  efforts,  which  can  have 
substantial  direct  and  indirect  cost  implications.  This  question  is  the  focus  of  this  report  of 
the  preliminary  conclusions  based  on  an  ongoing  qualitative  study. 
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We  began  this  study  seeking  to  understand  what  influences  the  successful  transition 
of  commercial  off-the-shelf  (COTS)  technologies  to  the  warfighter,  focusing  on  the  Joint 
Capabilities  Technology  Demonstration  (JCTD)  office  as  a  successful  case  study.  In  the 
course  of  our  investigation,  we  noted  shifts  in  organization  structure,  goals,  and  business 
processes  of  the  JCTD  office  resulting  from  responses  to  the  wars  in  Iraq  and  Afghanistan. 
Further  exploration  indicated  that  these  shifts  were  not  unique  to  the  JCTD  office  but  that 
the  shifts  we  observed  were  one  example  of  many  adaptive  solutions  to  changing  needs 
faced  by  the  DoD  acquisition  community.  In  order  to  better  understand  technology  transition 
in  the  current  context  and  in  accordance  with  a  grounded  research  approach,  we  adapted 
our  analysis  plan  to  focus  on  what  drives  successful  adaptation  (Howard-Grenville,  Golden- 
Biddle,  Irwin,  &  Mao,  2011;  Corbin  &  Strauss,  2008;  Lofland,  Snow,  Anderson  &  Lofland, 
2006).  This  report  is  based  on  our  initial  collection  and  analysis  of  archival  and  interview 
data.  We  are  continuing  to  collect  data  through  interviews  and  document  searches,  following 
a  process  of  theoretical  sampling  (Locke,  2001;  Clarke,  2005)  selecting  subjects  and 
documents  to  elaborate  on  the  concepts  reported  here. 

Since  2001  and  2003,  respectively,  U.S.  engagements  in  Afghanistan  and  Iraq  have 
highlighted  gaps  in  certain  capabilities:  U.S.  warfighters  were  not  always  equipped  for  the 
unique  challenges  they  faced  under  unanticipated  scenarios.  This  was  evidenced  by 
casualties  incurred  and  the  submission  of  more  than  7,000  urgent  need  statements 
(Gansler,  2009).  As  these  conflicts  ensued,  more  than  20  organizations  and  a  variety  of 
business  process  changes  emerged  to  meet  warfighter  needs.  This  situation,  and  the 
responses  to  it,  are  the  focus  of  the  widely  cited  “Gansler  report”  (2009),  which  forms  a 
context  for  this  study.  The  Gansler  report  stated,  “The  essence  of  the  problem  is  the  need  to 
field  militarily  useful  solutions  faster,”  and  “the  reality  is  that  the  Department  is  not  geared  to 
acquire  and  field  capabilities  in  a  rapidly  shifting  threat  environment”  (2009,  p.  viii).  The 
Gansler  report  concluded  that  the  ad  hoc  organizations  and  effective  processes  that 
emerged  to  meet  the  unanticipated  needs  of  U.S.  forces  in  Iraq  and  Afghanistan  should  be 
consolidated,  codified,  and  institutionalized.  This  conclusion  is  frequently  interpreted  as 
criticism  of  the  extant  acquisition  process  and  used  to  justify  further  expansion  of  ad  hoc 
solutions  (see,  for  example,  Warfighter  Support:  DoD’s  Urgent  Needs  Processes  Need  a 
More  Comprehensive  Approach  and  Evaluation  for  Potential  Consolidation,  GAO,  2011). 

In  accordance  with  what  is  formally  termed  an  “entrepreneurial  mindset”  (Haynie, 
Shepherd,  Mosakowski,  &  Earley,  2010),  we  reframe  this  interpretation  and  seek  to 
contribute  to  positive  changes  in  U.S.  defense  acquisition  through  an  analysis  based  on  it. 
Specifically,  we  explore  the  implications  to  DoD  acquisition  from  “standing  up  more  than  20 
ad  hoc  offices,  agencies,  task  forces,  funds,  and  other  organizations  to  respond  and  fulfill 
these  diverse  needs”  (Gansler,  2009)  and  the  problem-solving  these  entities  engaged  in  to 
emerge  as  an  exemplary  case  of  organizational  adaptation  to  unexpected  changes.  When 
conducting  qualitative  case  studies,  researchers  should  “go  for  extreme  situations,  critical 
incidents  and  social  dramas  ...  where  the  progress  is  transparently  observable”  (Pettigrew, 
1990,  p.  275).  Given  the  tremendous  size  and  bureaucratic  nature  of  the  DoD,  the  vital  role 
of  acquisition  on  the  organization’s  outcomes,  and  the  sudden  and  unpredictable  external 
change  presented  by  the  September  2001  attacks  and  subsequent  U.S.  engagements  in 
Afghanistan  and  Iraq,  we  view  the  acquisition  community’s  response  as  an  extreme  case, 
justifying  focused,  qualitative  exploration. 

Furthermore,  we  argue  that  reframing  the  Gansler  report  (2009),  to  view  the 
response  as  an  exemplary,  positive  case,  highlights  a  heretofore  under-appreciated  skill  set, 
at  which  the  DoD  may  excel.  Based  on  our  reframing  and  research  on  organizational 
routines,  dynamic  capabilities,  learning,  and  change,  we  examine  the  cost  and  benefits  of 
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investments  in  this  skill  set  and  other  business  capabilities.  Management  scholars  use  the 
term  capability  to  refer  to  a  high-level,  patterned  and  repetitious  routine  that  confers  a  set  of 
decision  options  for  producing  outputs  (Winter,  2003,  p.  991).  In  this  report,  we  will  use  the 
term  organizational  capability  to  distinguish  this  concept  from  the  concept  of  a  military 
capability,  which  is  perhaps  more  familiar  to  our  audience. 

This  report  proceeds  as  follows.  First,  we  ground  the  study  by  describing  the 
organizational  context  of  DoD  acquisition  and  the  events  that  resulted  in  recognition  of  the 
need  for  rapid  fielding.  Next,  we  analyze  and  reframe  the  2009  Gansler  report.  Then,  we 
describe  the  case  of  the  JCTD  and  our  methods  for  analyzing  it.  We  explore  the  potential 
costs  and  benefit  implications  of  different  approaches  to  securing  adaptive  business 
responses.  We  conclude  by  summarizing  our  preliminary  analysis  and  describing  the  next 
steps  in  our  ongoing  study. 

Defense  Acquisition  and  the  Shock  of  September  2001 

Acquisition  is  big  business.  Each  year,  the  DoD  spends  over  $100  billion  for 
research,  development,  procurement,  and  support  of  weapon  systems.  Acquisition  is  also  a 
rule-intensive  business.  In  addition  to  myriad  laws  governing  federal  acquisition  in  the  U.S., 
a  plethora  of  regulations  specify  how  to  accomplish  the  planning,  review,  execution,  and 
oversight  of  defense  acquisition  programs,  large  and  small,  sole-source  and  competitive, 
military  and  commercial.  Due  in  some  part  to  the  large  size  and  many  rules  associated  with 
defense  acquisition,  the  organizations  responsible  for  these  activities  tend  to  be  large  and 
rule-intensive  themselves,  reflecting  the  kinds  of  centralized,  formalized,  specialized,  and 
oversight-intensive  forms  corresponding  to  the  classic  “machine  bureaucracy”  from 
organization  theory.  The  problem  is,  this  classic  organizational  structure  is  well  known  to  be 
exceptionally  poor  at  responding  to  change.  In  the  context  of  military  transformation,  such  a 
problem  should  be  clear  and  compelling.  But  which  superior  organizational  approaches  are 
available  to  acquisition  leaders  and  policymakers?  What  evidence  supports  claims  of 
superiority  for  one  organizational  approach  versus  another?  Questions  such  as  these  are 
difficult  to  answer  through  most  research  methods  employed  to  study  organizations  (e.g., 
case  studies,  surveys,  etc.). 

Defense  acquisition  has  been  characterized  by  frequent  and  extensive  critique  and 
reform  over  the  past  50  years  leading  at  least  one  author  to  argue  that  “the  only  constant  in 
the  military’s  acquisition  system  is  the  continuous  reform”  (Rasche,  2011).  However,  driven 
by  the  changing  demands  of  warfighters,  the  commercial  rate  of  technological  development, 
and  defense  budget  constraints,  the  nature  and  speed  of  change  in  the  acquisition  system 
has  intensified  over  the  past  decade.  “Today’s  adversaries  are  changing  their  tactics, 
techniques,  and  procedures  at  an  accelerated  pace,  heightening  the  need  for  U.S.  forces  to 
respond  rapidly  to  new  threats”  (Gansler,  2009).  We  briefly  summarize  key  reformation 
events  of  the  past  two  decades  below. 

In  1993,  then  Vice  President  Al  Gore’s  Creating  a  Government  that  Works  Better 
and  Costs  Less:  The  Gore  Report  on  Reinventing  Government  sought  to  reduce 
government  waste  and  inefficiency,  calling  upon  the  DoD  acquisition  community  to  simplify 
procurement,  eliminate  regulatory  burden,  and  rely  to  a  greater  degree  on  the  commercial 
marketplace.  The  Clinton  administration  was  oriented  toward  “reinventing  government”  by 
improving  government  processes,  including  procurement.  Secretary  of  Defense  Leslie  Aspin 
voiced  his  concerns  that  acquisition  program  costs  and  schedule  problems  would  threaten 
the  ability  of  the  military  Services  to  continue  to  acquire  the  newest  technologies  that  had 
performed  so  well  during  the  Persian  Gulf  War.  Aspin  proposed  a  “resource  strategy”  to 
allow  the  DoD  to  afford  the  best  technology  in  a  times  of  austerity. 
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Shortly  thereafter,  Secretary  of  Defense  William  Perry  released  the  memo  “A 
Mandate  for  Change,”  which  called  for  a  cultural  change  within  the  DoD,  shifting  the  DoD’s 
focus  from  the  acquisition  process  to  its  outcome  in  the  field  and  asserting  that  the  major 
obstacles  to  positive  change  were  internal.  Acquisition  reform  continued  under  the 
leadership  of  Secretary  of  Defense  William  Cohen,  who,  in  a  1997,  expressed  the 
importance  of  continuing  to  reform  the  way  the  DoD  did  business,  demanding  that  the 
department  must  be  “lean,  agile,  and  focused  as  our  warfighters.”  The  report’s  main 
assertion  was  that  overhead  and  support  activities  be  reduced  and  reallocated  to  warfighters 
in  light  of  new  threats  and  constrained  budgets.  In  2000,  “The  Road  Ahead:  Accelerating  the 
Transformation  of  the  Department  of  Defense  Acquisition  and  Logistics  Processes  and 
Practices”  detailed  the  Revolution  in  Business  Affairs  (RBA),  which  called  for  best  practices 
from  the  private  sector  to  be  implemented  in  a  Revolution  in  Military  Affairs  (RMA).  The 
report  argued  that 

the  Department  continues  to  rely  on  acquisition  processes,  organizations  and 
infrastructure  largely  developed  in  the  years  following  World  War  II  [and] 
continues  to  face  a  limited  investment  budget,  and  squeezed  by  increased 
operations  and  support  costs  from  aging  weapons  systems.  (Gansler,  2000) 

On  September  10,  2001,  Secretary  of  Defense  Donald  Rumsfeld  gave  a  speech  in 
which  he  expressed  his  determination  to  save  the  Pentagon  from  itself.  The  Secretary 
claimed  that  the  Pentagon  bureaucracy  was  the  “serious  threat”  to  national  security,  but  he 
clarified,  saying,  “Not  the  people,  the  processes.  Not  the  civilians,  but  the  systems.  Not  the 
men  and  women  in  uniform,  but  the  uniformity  of  thought  and  action  that  we  too  often 
impose  on  them.”  Rumsfeld’s  vision  for  reform  included  commercial  outsourcing  of  functions 
not  directly  related  to  warfighting  to  save  money,  streamlining  the  system  development 
process  to  match  the  private  sector’s,  and  retaining  a  quality  workforce  within  the  military 
forces  and  acquisition  community.  Immediately  after  Rumsfeld’s  call,  the  events  of 
September  11th  occurred,  along  with  the  subsequent  wars  in  Iraq  and  Afghanistan.  These 
soon  highlighted  gaps  in  the  DoD’s  ability  to  rapidly  deploy  solutions  to  its  warfighters  facing 
their  new  scenarios  and  problems. 

In  both  Afghanistan  and  Iraq,  the  rapid  adaptation  of  enemy  capabilities  highlighted 
the  need  for  rapid  response  by  the  acquisition  community.  The  use  of  improvised  explosive 
devices  (lEDs)  in  Iraq  is  a  frequently  cited  example  of  enemy  forces  exploiting  “capability 
gaps  in  the  technology,  systems,  and  equipment  used  by  U.S.  forces”  (GAO,  2011). 
Combatant  commands  submitted  more  than  7,000  statements  for  urgent  solutions,  resulting 
in  the  eventual  creation  of  “over  20  ad  hoc  offices,  agencies,  task  forces,  funds  and  other 
organizations  to  meet  warfighter  needs”  (Gansler,  2009). 

The  Gansler  Report 

In  2009,  the  Defense  Science  Board’s  Task  Force  on  the  Fulfillment  of  Urgent 
Operational  Needs  published  a  report  known  widely  as  the  Gansler  report,  which  analyzed 
the  DoD’s  rapid  acquisition  process.  The  core  finding  of  the  report  was  that  major 
institutional  changes  needed  to  be  made  to  the  existing  DoD  acquisition  process.  The  report 
asserted  that  “rapid”  is  counter  to  the  current  acquisition  workforce  culture  and  that  the 
current  ad-hoc  system  is  not  sustainable  and  will  not  create  a  permanent  solution. 
Furthermore,  the  report  cited  institutional  barriers  (people,  funding,  and  processes)  as 
powerful  inhibitors  to  successful  rapid  acquisition  within  the  DoD.  Thus,  the  report  argued 
that  not  all  DoD  needs  can  be  met  by  the  same  acquisition  process  and  that  the  DoD  must 
create  and  codify  a  separate  “rapid”  process. 
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According  to  the  Gansler  report  (2009),  although  field  commanders  were  resourceful 
in  acquiring  local  solutions,  the  enemy’s  new  tactics  exploited  the  DoD’s  inability  to  rapidly 
field  new  capabilities.  The  Gansler  report  did  recognize  the  efforts  of  the  acquisition 
community,  stating,  for  example,  “It  is  hard  to  criticize  the  industrious  nature  of  those  in  the 
Department  who  have  made  something  happen  when  urgent  needs  have  been  presented” 
(Gansler,  2009,  p.  9).  However,  its  overall  perspective  and  its  interpretation  in  subsequent 
citations  is  a  largely  critical  call  for  reform:  “These  approaches  do  not  offer  a  long-term 
solution”  (Gansler,  2009,  p.  9).  In  particular,  the  report  highlighted  the  ad  hoc,  work-around 
nature  of  the  solutions,  noting  that  “numerous  rapid  reaction  programs  and  organizations 
have  been  established  in  recent  years  to  respond  to  combatant  commander  needs — 
processes  that  work  within  and  around  the  traditional  system  to  get  solutions  into  the  field” 
(Gansler,  2009,  p.  6),  and  citing  a  lack  of  institutional  changes  to  organize,  formalize,  and 
codify  the  ad  hoc  approaches  as  evidence  of  continued  failure. 

By  and  large,  the  Gansler  report  (2009)  represented  the  breadth  of  criticisms  of  the 
DoD  rapid  acquisition  process  and  its  ad  hoc  entities  since  their  emergence  shortly  after  the 
invasion  of  Iraq.  More  recent  assessments  offer  similar  criticisms.  The  GAO’s  (2011)  report 
to  congressional  committees  in  2011  titled  Warfighter  Support:  DoD’s  Urgent  Needs 
Processes  Need  a  More  Comprehensive  Approach  and  Evaluation  for  Potential 
Consolidation  identified  at  least  31  separate  entities  that  manage  urgent  acquisition  needs. 
The  report  claimed  that  the  numerous  points  through  which  a  warfighter  may  submit  a 
request  for  an  urgent  need  is  an  example  of  redundancy  and  inter-agency  overlap.  The 
GAO  (2011)  asserted  that  the  DoD  does  not  have  a  comprehensive  policy  for  how  urgent 
needs  are  to  be  addressed,  lacks  visibility  over  the  full  range  of  its  urgent  needs  efforts,  has 
no  senior-level  focal  point  to  lead  the  department’s  efforts  to  fulfill  urgent  needs,  and  has  not 
evaluated  opportunities  for  consolidation,  resulting  in  unnecessary  costs.  The  GAO  (201 1 ) 
ultimately  attributed  the  need  for  the  many  ad  hoc  processes  that  currently  exist  to  a  failure 
of  the  DoD  to  predict  change  in  the  external  environment,  saying,  “The  department  had  not 
anticipated  the  accelerated  pace  of  change  in  enemy  tactics  and  techniques  that  ultimately 
heightened  the  need  for  a  rapid  response  to  new  threats  in  Afghanistan  and  Iraq.” 

The  conclusions  and  tone  of  these  reports  appear  critical  of  the  so-called  ad  hoc 
solutions.  For  example,  the  Gansler  report  noted,  “While  these  programs  have  produced 
significant  successes,  their  ad-hoc,  one  of  a  kind  nature  has  created  a  different  set  of 
problems.  They  rely  on  learning  on  the  job  with  little  emphasis  on  support  training  and 
sustainment”  (Gansler,  2009,  p.  6).  Perhaps  unsurprisingly,  given  the  bureaucratic  nature 
and  culture  of  the  DoD,  the  reports  call  for  centralization,  formalization,  and  codification  to 
correct  the  problem  presented  to  the  DoD  organization  by  the  ad  hoc  organizations  and 
processes.  Indeed,  we  have  previously  suggested  that  the  DoD  has  a  propensity  or 
preference  toward  such  centralization,  to  its  own  detriment  (Dillard,  2005).  Given  the  current 
nature  and  culture  of  the  DoD,  the  survival  of  rapid  or  urgent  fielding  capabilities  may  indeed 
depend  on  some  form  of  the  solutions  recommended  in  these  reports.  However,  we  argue  it 
is  important  to  note  that  in  framing  ad  hoc  responses  as  a  problem  and  then  offering  a 
solution,  these  reports  fail  to  address  the  institutional  and  cultural  environment,  which  they 
argue  cannot  sustain  innovation.  Of  perhaps  greater  concern,  it  is  possible  that  enacting  the 
recommendations  of  the  reports  without  full  consideration  of  the  value  of  the  ad  hoc  problem 
solving  that  occurred  and  the  costs  associated  with  building  a  “dynamic  capability,”  the  DoD 
may  eventually  lose  a  valuable  source  of  business  process  and  organizational  innovation 
and  adaptation  and/or  may  overinvest  in  a  costly  organizational  solution,  when  a  less  costly 
alternative  may  suffice. 


m  khsiikm 
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Research  Context:  Framing  Rapid  Fielding 

We  situate  this  study  in  a  reframing  of  the  widely  cited  Gansler  report  of  2009.  Our 
reframing  is  conducted  in  the  spirit  of  the  accepted  wisdom  that  creative  solutions  often 
require  “thinking  out  of  the  box”  or  “lateral  thinking”  (De  Bono,  1967),  which  we  equate  more 
formally  with  adopting  an  entrepreneurial  mindset — described  below — and  guided  by  a 
research  approach  based  on  frame  analysis.  We  undertake  this  exploration  not  to  argue 
against  specific  recommendations  of  the  Gansler  report,  but  rather  because  we  believe  that 
a  problem  of  such  persistence  and  consequence  deserves  considered  reflection  from 
multiple  perspectives. 

Research  Framework 

An  entrepreneurial  mindset  is  the  ability  to  “think  differently,”  to  sense,  act,  and 
mobilize  under  uncertain  conditions  (Haynie  et  al.,  2010).  Adaptive  thinking  hinges  on  “the 
ability  to  be  dynamic,  flexible,  and  self-regulating  in  one’s  cognitions”  (Haynie  et  al.,  2010,  p. 
218)  and  is  of  fundamental  importance  to  entrepreneurs  or  others  facing  uncertain  task 
environments.  Adaptive  thinking  is  dependent  on  metacognitive  processes — thinking  about 
thinking — which  enable  individuals  to  think  beyond  existing  heuristics  and  knowledge 
structures  in  order  to  be  adaptable.  A  metacognitive  strategy  refers  to  the  mental  framework 
formulated  by  an  individual,  through  which  to  evaluate  multiple,  alternative  responses  to 
processing  a  task.  Researchers  have  demonstrated  that  employing  a  metacognitive  strategy 
can  improve  the  outcome  of  problem  solving  by  helping  individuals  avoid  using  a  flawed 
approach  for  addressing  a  problem  (Staw  &  Boettger,  1990;  Haynie  et  al.,  2010). 

Drawing  on  these  arguments,  Haynie  et  al.  (2010)  argued  that  successful 
entrepreneurs  will  be  those  that  formulate  a  metacognitive  strategy  to  generate  alternative 
approaches  to  thinking  about  how  to  accomplish  tasks  in  ambiguous  environments.  In  other 
words,  entrepreneurs  who  succeed  will  be  those  who  can  develop  multiple,  alternative  ways 
of  thinking  about  a  problem.  We  approached  this  research  in  this  spirit,  seeking  an 
alternative  strategy  for  thinking  about  the  problem  of  acquisition  reform  in  order  to  evaluate 
possible  responses. 

A  metacognitive  strategy  requires  metacognitive  awareness,  that  is,  awareness 
concerning  one’s  own  thinking.  We  thus  undertook  an  examination  of  the  logic, 
assumptions,  and  links  between  these  and  the  conclusions  presented  in  the  Gansler  report. 
Our  examination  followed  the  norms  and  precepts  of  frame  analysis  as  developed  in 
organization  research  (Benford  &  Snow,  2000;  Creed,  Langstraat,  &  Scully,  2002). 

Frames  are  “action-oriented  sets  of  beliefs  and  meanings  that  inspire  and  legitimate 
that  activities  and  campaigns”  created  through  conversations  and  written  communication 
that  connect  events  and  experiences  (Benford  &  Snow,  2000).  Core  framing  tasks  include 
diagnostic  framing,  the  identification  of  problems  and  causes;  and  prognostic  framing,  the 
articulation  of  a  proposed  solution.  Institutional  solutions  to  problems  result  when  recurring 
or  widespread  problems  are  theorized,  or  described  in  general  terms,  and  agreed  upon, 
pointing  to  a  particular  solution  (Suchman,  1995).  Following  Creed  et  al.  (2002),  we 
developed  a  signature  matrix  to  sort  the  idea  elements  found  in  the  Gansler  report  into 
categories  that  support  the  functions  of  interpretation,  argumentation,  punctuation, 
elaboration,  and  motivation.  This  allowed  us  to  discern  key  elements  of  the  frame  and 
consider  alternatives. 

The  Framing  of  the  Gansler  Report 

The  Gansler  report  (2009)  depicted  the  response  to  the  unanticipated  needs  of 
warfighters  in  Afghanistan  and  Iraq  as  evidence  that  the  DoD  cannot  respond  to  changing 
needs.  The  report  framed  the  emergence  of  many  organizations  and  the  lack  systematic, 
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codified  processes  as  evidence  of  failure,  and  problems,  which  must  be  corrected.  In 
particular,  the  report  highlighted  the  lack  of  sustainable  funding  for  ad  hoc  processes  as  a 
problem  for  which  the  solution  is  codification,  centralization,  and  formalization.  Although  this 
is  a  logical  solution  to  the  problem  as  framed  in  the  report,  an  alternate  frame  might  suggest 
other  possible  solutions. 

In  the  Gansler  report,  the  large  number  of  requests  to  meet  urgent  needs,  and  the 
highly  visible  problem  of  lEDs,  are  used  to  support  the  assertion  that  the  DoD  “lacks  the 
ability  to  rapidly  field  new  capabilities”  (2009).  The  text  of  the  report  includes  the  phrase  “in  a 
systematic  and  effective  way,”  linking  the  assertion  of  failure  and  a  lack  of  systematic 
processes  to  ineffectiveness.  This  depiction  is  further  linked  to  an  overall  presentation  of  the 
problem  or  the  diagnostic  frame;  the  lack  of  systematic  processes  makes  the  current 
solution  unsustainable,  and  as  the  problem  is  the  lack  of  systematic  processes,  the  solution 
is  therefore  the  creation  of  a  systematic,  codified  process  in  a  formal,  centralized 
organization.  The  latest  update  of  the  Joint  Capabilities  Integration  and  Development 
instruction,  CJCSI  31 70.01  H  (2012),  already  reflects  some  implementation  of  this 
recommendation. 

Although  some  successful  outcomes  result  from  ad  hoc  organizations  and  business 
processes,  recognition  of  achievements  are  followed  by  critiques  of  the  processes  that 
achieved  them.  Variation  is  presented  as  redundant  and  costly.  Ad  hoc  problem  solving  is 
not  systematic  or  codified  (and  linked  to  ineffective  and  unsustainable).  Workarounds, 
although  recognized  as  necessary,  are  depicted  as  “disjointed”  (linked  to  unsystematic  and 
ineffective).  For  example, 

Over  the  past  five  years  there  have  been  many  success  stories  and  lessons 
Learned.  ...  However,  in  the  larger  picture,  the  DoD  has  not  made  major, 
institutional  changes  in  budgeting  and  acquisition  essential  to  posture  itself 
for  the  ongoing  hybrid  warfare  reality.  DoD  is  not  systematically  prepared  to 
anticipate  and  respond  to  urgent  and  dynamically  changing  needs  that  will  be 
a  permanent  part  of  21st  century  operations. 

When  progress  is  noted,  it  (progress)  refers  to  codification,  as  in  this  example: 

The  Joint  Staff,  COCOMs,  and  the  Services  have  all  codified  in  directives 
new  processes  to  identify  urgent  needs  and  provide  rapid  responses.  Recent 
progress  includes  a  detailed  urgent  needs  process  memorandum  circulated 
by  the  Secretary  of  the  Navy  in  March  2009. 

The  arguments  of  the  report  support  the  recommendation  to  restructure  the 
organization  and  to  create  a  codified,  systematic  process  for  rapid  fielding.  This 
recommendation  is  consistent  with  the  bureaucratic  nature  and  culture  of  the  DoD  and  with 
past  routines  for  codifying,  reorganizing,  and  centralizing.  However,  a  reframing  of  the 
problem  allows  a  deeper  consideration  of  factors  mentioned  but  not  emphasized  in  the 
report  and  illuminates  heretofore  underemphasized  or  overlooked  implications  of  the  report’s 
recommendations. 

An  Alternate  Perspective 

We  explored  the  question  “What  is  the  most  cost  effective  means  of  achieving  the 
dynamic  and  adaptive  business  capabilities  DoD  seems  to  require?”  We  began  by  reframing 
the  Gansler  report.  A  summary  of  our  analysis  and  reframing  is  shown  in  Table  1 .  In  our 
reframing,  we  considered  the  establishment  of  20  (and  eventually  more  than  30) 
organizational  entities  over  a  period  of  a  few  years  and  their  development  of  associated 
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business  models  and  processes  to  be  an  amazing  adaptive  response  to  an  external  shock 
by  a  bureaucratic  organization,  which  would  be  expected  to  be  hampered  by  severe  inertia. 


Table  1 .  Framing  of  the  Gansler  Report 


Focal  event 

Warfighters  in  Afghanistan  and  Iraq  have  unanticipated  equipment  needs 

Gansler  Frame 

Representative  Quote 

Alternate  Frame 

Depiction 

DoD  has  not 

responded/cannot 

respond. 

DoD  lacks  the  ability  to  rapidly 
field  new  capabilities  to  the 
warfighter  (in  a  systematic  and 
effective  way). 

Acquisition  community 
responded. 

Punctuation: 

What  is  the 
problem? 

Current  rapid  fielding 
process  is 
unsustainable. 

The  essence  of  the  problem  is  the 
need  to  field  militarily  useful 
solutions  faster. 

Adapting  (business  organization) 
to  changing  environment. 

Current  approaches  to  implement 
rapid  responses  to  urgent  needs 
are  not  sustainable. 

Current  process  is  an  example  of 
a  valuable,  periodically  utilized 
skill-set. 

Elaboration: 

What  factors 
contribute? 

Variation  is 
redundant  and 
costly. 

The  procedures  these 
organizations  have  developed  ... 
vary  across  the  DoD  ...  definitions 
and  regulations  that  apply  to  the 
processes  vary  [and  words]  . . .  are 
sometimes  used  in  conflicting  and 
overlapping  ways. 

Variation  is  a  necessary 
component  of  change. 

Ad  hoc  problem 
solving  is 
problematic. 

Their  ad  hoc,  one-of-a-kind  nature 
has  created  a  different  set  of 
problems.  They  rely  on  learning 
on  the  job  with  little  emphasis  on 
support,  training,  and 
sustainment. 

Ad  hoc  problem  solving  is  a  “low 
cost”  skill  set. 

Workarounds 
contradict  the 
institution. 

All  also  utilize  workarounds  ...  to 
sidestep  traditional  acquisition  and 
fielding  process,  but  these  are 
generally  disjointed. 

Workarounds  allow  creativity 
within  a  bureaucracy. 

Formalization, 
codification,  and 
consolidation  result 
in  sustainability. 

DoD  needs  to  codify  and 
institutionalize  “rapid”  acquisition 
processes  and  practices. 

Codification  is  costly.  The  full 
value  lies  in  the  knowledge 
gained  through  the  process, 
gaining  full  value  requires 
collaboration. 

Motivation: 

What  action 
should  be 
taken? 

Undertake  structural 
reforms  to 
institutionalize  a 
specific  solution. 

The  Secretary  of  Defense  should 
establish  a  new  agency. 

Evaluate  costs/benefits  of  ad  hoc 
solutions  and  seek  solutions  that 
retain  diverse  skill  sets. 

Our  perspective  is  not  without  precedent,  even  within  the  DoD.  In  a  201 1  report, 
Lessons  Learned  From  Rapid  Acquisition:  Better,  Faster,  Cheaper?,  Colonel  Robert  A. 
Rasch  examined  the  impacts  of  wartime  acquisition  initiatives  on  the  DoD  acquisition 
systems.  Rasch  framed  the  continual  reform  of  DoD  acquisition  as  a  possible  indicator  of 
positive  adaptive  change.  Perhaps  best  known  is  the  large  scale  and  rapid  acquisition  of  at 
least  7,000  Mine  Resistant  Ambush  Protected  (MRAP)  vehicles  in  just  over  two  years.  The 
need  for  MRAP  vehicles  was  initially  articulated,  in  February  of  2005  by  Marines  who 
needed  protection  from  lEDs,  RPGs,  and  small-arms  fire.  The  need  was  met  through  a 
variety  of  ad  hoc  solutions  involving  innovative  adaptations  to  standard  processes  for 
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establishing  requirements,  evaluating  progress,  and  contracting.  This  instance  is  cited  as  an 
exemplary  outcome  in  GAO  reports  (GAO,  2009). 

Viewing  this  response  above  as  a  successful  solution  suggests  a  reconsideration  of 
the  definition  of  the  problem.  The  Gansler  report  (2009)  is  clearly  focused  on  the  immediate 
need  for  rapid  fielding,  as  tasked,  and  our  reframing  should  not  be  viewed  as  a  criticism  of 
those  efforts.  However,  when  given  the  luxury  of  reflective  consideration  afforded  a  research 
project  (as  opposed  to  the  task  specific  demands  facing  a  decisively  engaged  military  force), 
the  context  of  the  organization,  past  attempts  at  reform  and  an  environment  characterized 
by  unpredictable  events,  suggest  a  broader  and  persistent  need  for  business  adaptability. 
We  reframe  the  problem  in  terms  of  this  broader  need:  The  DoD  must  adapt  its  business 
model  and  processes  to  meet  unpredictable  demands  from  the  external  environment.  This 
need  is  recognized  in  the  Gansler  report: 

The  global  landscape  has  changed  the  national  security  environment, 
demanding  the  ability  to  rapidly  access  and  field  capabilities  from  any  source. 
Agile  adversaries  are  taking  advantage  of  important,  globally  available 
technologies  by  rapidly  creating  and  fielding  highly  effective  weapons. 
Moreover,  the  nation  faces  a  vast  range  of  potential  contingencies  around  the 
world.  ...  This  set  of  circumstances  calls  for  rapid  adaptation  on  the  part  of 
the  United  States  as  well — adaptation  of  tactics,  techniques,  and  procedures 
[emphasis  added]  as  well  as  the  ability  to  field  new  [warfighting]  capabilities 
on  a  timeframe  unfamiliar  to  the  bureaucratic  processes  that  dominate 
acquisition  in  the  Department  of  Defense  today.  (2009,  p.  3) 

However,  the  overriding  focal  problem  highlighted  by  the  framing  of  the  Gansler 
report  is  the  need  for  a  rapid  fielding  capability.  Reframing  the  problem  as  we  have  done 
suggests  a  reconsideration  of  the  role  and  value  of  variation,  ad  hoc  problem  solving,  and 
codification.  The  Gansler  report  frames  these  factors  as  contributors  to  the  problem.  In  our 
reframing,  we  considered  the  role  of  variation  as  precursor  to  change,  workarounds  as  a 
mechanism  for  allowing  creativity  within  a  bureaucracy,  and  the  benefits  of  codification  as 
deriving  from  the  process  of  articulation  and  clarification  as  much  as  (or  even  more  than) 
from  written  output.  Our  reframing  suggests  a  need  to  evaluate  the  costs  and  benefits  of  ad 
hoc  problem  solving  versus  codified  business  capabilities  and  to  seek  overall  solutions  that 
most  efficiently  support  the  business  adaptability  in  an  unpredictable  environment. 

Research  Approach  and  Methods 

We  began  our  study  of  the  JCTD  case  with  the  question  of  what  best  influences  the 
successful  transition  of  commercial-off-the-shelf  (COTS)  technologies  to  the  warfighter. 
During  our  initial  investigation,  we  noted  shifts  in  organization  structure,  goals,  and  business 
processes  of  the  JCTD  office  in  response  to  the  wars  in  Iraq  and  Afghanistan.  In  accordance 
with  a  grounded  research  approach  (Howard-Grenville  et  al. ,  2011;  Corbin  &  Strauss,  2008; 
Lofland  et  al.,  2006),  we  adapted  our  analysis  plan  to  focus  on  how  the  organization  was 
adapting  to  change.  This  report  is  based  on  our  initial  collection  and  analysis  of  archival  and 
interview  data.  The  organization  is  once  again  adapting  as  the  need  for  rapid  fielding  in 
Afghanistan  and  Iraq  diminish,  and  our  analysis  to  this  point  must  thus  be  considered 
preliminary.  We  are  continuing  to  collect  data  through  interviews  and  document  searches, 
following  a  process  of  theoretical  sampling  (Locke,  2001;  Clarke,  2005). 

We  began  this  study  with  a  review  of  literature  related  to  the  JCTD  office  and  the 
evolution  of  defense  acquisition  processes.  We  also  conducted  a  round  of  exploratory 
interviews  with  subject  matter  experts  in  the  JCTD  office.  These  were  informal,  unstructured 
interviews,  designed  to  familiarize  us  with  the  history,  operations,  and  evolution  of  the  office. 
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We  encouraged  experts  to  elaborate  on  these  topics  and  took  detailed  notes.  In  the  course 
of  the  initial  data  collection,  we  noted  an  apparent  and  deliberate  shift  had  occurred  in  the 
mission  of  the  JCTD  office  in  recent  years,  from  demonstrating  advanced  militarily  useful 
concepts  with  promising  technologies  towards  rapid  fielding  of  materiel  and  the  importance 
of  ad  hoc  problem  solving. 

We  collected  additional  data  from  two  sources:  a  “snowballing”  Google  search  and 
the  Internet  Archive  (Nardon  &  Aten,  2008;  Aten,  2010).  On  Google,  we  searched  for  all 
pages  and  documents  with  JCTD  or  ACTD  and  the  word  technology  in  the  title  from  the  year 

2000  to  the  present  and  saved  each  as  a  PDF,  yielding  more  than  2,000  pages.  We  then 
followed  links  to  identify  additional  pages  and  documents,  yielding  an  initial  247  saved 
PDFs.  We  scanned  all  of  the  documents  and  excluded  documents  such  as  glossary  pages, 
descriptions  of  acronyms,  and  descriptions  and  press  releases  related  to  particular  JCTDs. 
This  yielded  a  dataset  that  included  presentation  slides,  JCTD  announcements  and  policies, 
and  descriptions  of  the  organization. 

Next,  we  collected  data  from  the  Internet  Archive  (2009),  “a  non-profit  organization 
that  was  founded  to  build  an  Internet  library,  with  the  purpose  of  offering  permanent  access 
for  researchers,  historians,  and  scholars  to  historical  collections  that  exist  in  digital  format.” 
The  Internet  Archive  is  searchable  by  URL  with  a  search  resulting  in  a  list  of  hyperlinks  to 
web  pages  for  the  specified  URL,  by  date,  that  are  included  in  the  archive.  Thus,  one  can 
view  web  pages  of  an  organization  as  they  existed  for  a  particular  year  in  the  past.  The 
archive  for  the  ACTD  and  JCTD  was  intact,  with  multiple  instances  captured  every  year  from 

2001  to  the  present.  We  reviewed  one  web  page  per  year,  adding  instances  as  necessary 
when  we  noted  major  changes  to  ensure  that  we  did  not  miss  relevant  documents.  On  each 
page,  we  followed  links  and  printed  PDF  files  of  web  pages  and  documents  related  to  the 
evolution  of  the  JCTD  office.  We  selected  pages  and  documents  available  from  links  titled 
introduction,  guidelines,  Q&A,  links,  organization,  and  what’s  new.  Our  saved  documents 
included  conference  presentation  slides,  management  briefings,  procedures  and  guidelines, 
organization  charts,  and  the  text  of  speeches.  We  did  not  save  specific  JCTD  project 
descriptions,  glossary  pages,  or  point  of  contact  information  pages. 

We  organized  all  of  the  documents  by  year  and  imported  them  into  an  Nvivo 
qualitative  data  analysis  software  project.  We  used  Nvivo  to  code  the  data  into  broad 
categories  suggested  by  our  previous  analysis:  organization  structure,  business  model 
(mission/goals,  value  proposition,  measures),  technology  characteristics  (maturity  level,  use, 
customer),  and  process  characteristics  (requirements,  steps).  We  then  generated  reports 
allowing  us  to  view  examples  from  the  broad  categories  across  time. 

Research  Setting:  The  Joint  Capabilities  Technology  Demonstration  Office 

The  JCTD  program  began  in  1994  as  the  Advanced  Concepts  Technology 
Demonstration  (ACTD),  with  the  aim  of  more  rapid  prototyping  and  fielding  of  technology  for 
the  DoD  by  demonstrating  and  assessing  the  of  the  military  utility  of  a  technology.  Over  the 
18  years  since  its  inception,  the  overall  mission  of  the  program  has  remained  unchanged. 

History  and  Purpose 

In  the  late  1980s,  the  President’s  Blue  Ribbon  Commission  on  Defense,  also  known 
as  the  Packard  Commission,  was  charged  by  Executive  Order  12526,  in  which  President 
Reagan  asked  the  commission  to  conduct  a  defense  management  study  focusing  on  the 
budget  process,  the  procurement  of  systems,  the  legislative  oversight,  and  intra-government 
organizational  arrangements  in  regard  to  defense.  Among  other  things,  the  report  indicated 
a  high  need  for  prototyping.  The  report  stated  that 
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a  high  priority  should  be  given  to  building  and  testing  prototype  systems  and 
subsystems  before  proceeding  with  full-scale  development.  This  early  phase 
of  R&D  should  employ  extensive  informal  competition  and  use  streamlined 
procurement  processes.  It  should  demonstrate  that  the  new  technology  under 
test  can  substantially  improve  military  capability,  and  should  as  well  provide  a 
basis  for  making  realistic  cost  estimates  prior  to  a  full  scale  development 
decision.  This  increased  emphasis  on  prototyping  should  allow  us  to  “fly  and 
know  how  much  it  will  cost  before  we  buy.” 

The  Packard  Commission  report,  as  well  as  several  other  Defense  Science  Board 
reports,  led  to  the  establishment  of  the  ACTD.  ACTDs  are  user-oriented  and  of  a  large 
enough  scope  to  establish  military  utility.  During  the  ACTD,  the  users  (the  warfighters) 
determine  whether  they  will  begin  acquisition  of  the  new  technology.  The  ACTDs/JCTDs 
serve  the  Combatant  Commands  (COCOM)  by  fulfilling  capability  gaps  the  Services  may  not 
view  as  mission-critical  but  that  the  COCOMs  are  nonetheless  requesting. 

In  2006,  the  ACTD  became  the  JCTD.  Although  the  core  staff  and  office  remained 
the  same,  the  name  change  brought  with  it  a  change  in  focus;  there  was  a  shift  to 
emphasizing  the  fulfillment  of  capabilities  and  an  added  emphasis  on  transitioning  new 
technologies  to  the  field  for  sustained  use.  Despite  some  changes  in  management,  name, 
and  participation  of  various  agencies,  the  organizational  structure  of  the  ACTD/JCTD  has 
remained  fairly  constant.  An  ACTD/JCTD  is  jointly  sponsored  and  managed  by  a  supporting 
user  (the  military)  and  the  technology  developer.  Approval  of  ACTD/JCTDs  is  given  by  the 
Deputy  Under  Secretary  for  Advanced  Systems  and  Concepts  (DUSD  [AS&C]).  The 
ACTD/JCTD  program  maintains  a  significant  cross-service,  cross-agency  involvement  with  a 
heavy  focus  on  joint  operations  and  COCOM  participation.  In  September  2009,  the  DoD 
established  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Rapid  Fielding 
(ODASD[RFj).  Sometime  shortly  after  its  establishment,  the  ODASD(RF)  was  designated  as 
the  overseeing  agency  of  the  JCTD  program. 

Although  the  personnel  and  management  remained  the  same,  the  JCTD  program 
claims  to  be  implementing  a  new  and  enhanced  business  process  to  better  meet  the  DoD’s 
transformational  goal  of  becoming  capabilities  based.  JCTDs  focus  directly  on  the  COCOM’s 
most  critical  warfighter  needs  and  proved  a  faster,  more  agile  and  integrated  joint  response 
to  emerging  asymmetrical  threats.  JCTDs  emphasize  increased  upfront  transition  planning, 
provision  for  a  higher  level  of  OSD  funding  during  the  first  two  years,  and  bridge  funding 
from  Budget  Activity  Four  for  those  projects  that  demonstrate  compelling  joint  military  utility. 
In  the  move  from  ACTD  to  JCTD  the  program  eliminated  several  of  the  review  processes, 
such  as  the  so-called  Breakfast  Club,  and  limited  the  involvement  of  the  Joint  Chiefs.  The 
program  was  redirected  to  focus  more  on  capabilities  and  transitioning  the  new  capabilities 
but  also  on  rapid  fielding;  the  ACTD  program  saw  a  50-60%  transition  rate,  as  the  JCTD 
program  is  seeing  an  80-90%  transition  rate. 

Technology 

An  important  part  of  considering  candidates  to  become  an  ACTD/JCTD  is  the 
technology  readiness  level  (TRL).  “Technology  maturity  is  a  measure  of  the  degree  to  which 
proposed  critical  technologies  meet  program  objectives;  and,  is  a  principal  element  of 
program  risk.”  The  DoD  Component  Science  and  Technology  (S&T)  Executive  directs  the 
technology  readiness  levels  and  determines  the  level  of  maturity  of  a  given  system. 

There  are  nine  TRL  levels,  each  representing  a  major  step  forward  in  the 
development  process  of  the  system.  ACTDs/JCTDs  are  largely  previously  proven 
technologies  that  will,  by  and  large,  have  a  TRL  of  7,  8,  or  9.  A  system  that  is  ranked  with  a 
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TRL  7  has  demonstrated  “an  actual  system  prototype  in  an  operational  environment.”  TRL  8 
is  assigned  to  technology  that  “has  been  proven  to  work  in  its  final  form  and  under  expected 
conditions.”  In  almost  all  cases,  this  TRL  represents  the  end  of  true  system  development. 
TRL  9  is  assigned  to  technology  “in  its  final  form”  and  that  has  been  proven  through 
successful  mission  operations. 

There  are  several  characteristics  for  which  ACTD  candidates  are  chosen: 
affordability,  interoperability,  sustainability,  and  potential  for  evolution.  The  affordability  of  a 
new  capability  was  viewed  from  the  perspective  of  the  total  ownership  cost,  to  see  whether 
the  cost  of  the  capability  throughout  its  life  cycle  would  hinder  its  eventual  inclusion  into  the 
regular  acquisition  process.  The  new  technology  or  capability  was  required  to  be 
interoperable  because  of  the  importance  of  implementing  the  technology  in  future 
operations.  The  new  systems  remain  in  the  field,  so  sustainability  was  a  crucial  aspect. 
Finally,  systems  and  capabilities  were  evaluated  based  on  their  potential  to  be  updated  as 
the  situation  or  threat  evolved. 


The  TRL  of  ACTDs  fluctuated  depending  on  the  type  of  system  and  the  level  of  risk 
that  managers  and  oversight  organizations  were  willing  to  take.  In  the  period  before  2003, 
projects  were  much  larger  and  assumed  more  risk  in  term  of  the  readiness  of  the 
technologies  (Global  Hawk  and  Predator).  The  Defense  Advanced  Research  Projects 
Agency  (DARPA)  was  once  one  of  the  largest  contributors  to  the  funding  of  ACTDs; 
however,  eventually  DARPA’s  involvement  in  the  program  waned,  and  so  too  did  the  large 
and  risky  nature  of  many  ACTDs. 


As  the  ACTD  transitioned  to  the  JCTD  and  as  time  went  on,  the  program  became 
more  focused  on  picking  “the  low  hanging  fruit”  in  the  sense  that  ACTDs  became  more 
focused  on  smaller  projects  that  assumed  less  risk.  This  has  also  been  attributed  to  the 
increased  focus  of  rapid  fielding  that  was  generated  by  the  wars  in  Iraq  and  Afghanistan. 
Figure  1  shows  the  relatively  steady  decline  in  the  average  estimated  costs  of  the 
ACTD/JCTD  projects  by  year  for  the  last  10  years.  The  decline  in  costs  coupled  with  the 
decline  in  the  average  length  is  evidence  that  lends  itself  to  the  notion  that  the  program  was, 
as  one  official  put  it,  focused  on  “getting  something  out  the  door  as  quickly  as  possible.” 


Figure  1 .  Average  Estimated  Costs  of  the  ACTD/JCTD  Projects  by  Year  for 

the  Last  10  Years 


More  recently  (in  last  few  months)  and  after  a  change  in  management,  the  JCTD  has 
encountered  criticism  for  its  increasing  aversion  to  risk,  which  was  generally  coming  from 
senior  leadership  of  the  program.  Also,  the  need  for  rapid  fielding  has  been  lessened  by  the 
ending  of  the  Iraq  War  and  the  winding-down  of  operations  in  Afghanistan.  Now,  there  is  an 
emerging  desire  to  shift  the  JCTD  back  to  its  original  style  of  bigger,  better  and  riskier  and  to 
adapt  once  more. 
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Dynamic  Capabilities  and  Ad  Hoc  Problem  Solving:  Pathways  to  Adaptability 

Although  the  political  environment  is  not  perfectly  analogous  to  the  business 
environment,  some  useful  comparisons  can  be  made.  The  shocks  of  9/11  and  enemy 
innovations  suggest  the  acquisition  community  is  facing,  and  will  continue  to  face,  a 
turbulent  environment.  Studies  of  organizations  operating  in  turbulent  environments  have 
focused  on  understanding  the  role  of  routines  in  change  and  adaptation.  Scholars  have 
argued  that  dynamic  capabilities,  or  the  ability  to  systematically  change  existing 
organizational  routines  are  a  key  to  success  (Teece,  2007).  However,  Winter  (2003)  argued 
that  the  costs  of  creating  dynamic  capabilities  may  not  be  justifiable  in  turbulent 
environments.  Winter’s  (2003)  argument,  along  with  a  recent  discussion  of  anticipated 
consequences  in  such  environments  (Selsky  et  al.,  2007),  suggests  that  ad  hoc  problem 
solving  may  be  an  effective  solution  for  adapting  to  change.  We  discuss  these  ideas  below. 

Understanding  organizational  adaption  and  change  is  a  key  focus  of  organizations 
scholars.  Organizational  routines  provide  one  avenue  for  exploring  how  organizations 
change  their  capabilities.  Organizational  routines  are  the  basic  components  of 
organizational  behavior  and  are  a  crucial  to  understanding  how  organizational  capabilities 
are  accumulated,  transferred,  and  applied  (Becker,  Lazaric,  Nelson,  &  Winter,  2005).  Thus, 
organizational  routines  provide  a  useful  starting  point  for  an  exploration  of  the  pathways  to 
organizational  adaptability.  The  discussion  below  draws  largely  from  Winter’s  (2003) 
“Understanding  Dynamic  Capabilities.” 

An  organizational  routine  is  highly  patterned,  repetitious  behavior  that  is  learned, 
founded  at  least  in  part  in  tacit  knowledge  and  directed  toward  specific  objectives.  Thus, 
behaviors  to  run  a  particular  production  line  to  produce  a  particular  product  constitute  a 
routine.  Organizational  improvisation  is  not  a  routine  because  it  is  dynamic,  one  of  kind,  and 
conscious  rather  than  patterned,  repetitions,  and  tacit  behavior.  An  organizational  capability 
is  a  high-level  routine  that  confers  upon  an  organization’s  management  a  set  of  decision 
options  for  producing  a  particular  type  of  output. 

Recent  research  on  strategy  in  rapidly  changing  environments  has  focused  on 
dynamic  capabilities  (Teece,  Pisano,  &  Shuen,  1997;  Eisenhardt  &  Martin,  2000).  Despite 
the  name,  dynamic  organizational  capabilities  are  based  on  routines  and  patterned, 
repetitious  behavior.  The  dynamic  refers  to  the  focus  of  the  routine.  Ordinary  organizational 
capabilities  are  operational  capabilities.  Those  organizational  capabilities  that  provide  value 
exhibit  technical  and  environmental  fit,  allowing  an  organization  to  “make  a  living”  by 
performing  a  particular  function  well  and  also  by  allowing  an  organization  to  succeed  within 
a  particular  environment,  respectively.  Dynamic  capabilities  are  organizational  capabilities 
that  extend,  modify,  or  create  ordinary  capabilities,  helping  organizations  shape  and  adapt 
to  the  environment,  achieving  evolutionary  fitness.  Dynamic  capabilities  involve  sensing  and 
shaping  opportunities  and  threats,  seizing  opportunities,  and  maintaining  competitiveness  by 
combining,  enhancing,  protecting,  and  reconfiguring  tangible  and  intangible  assets.  Zollo 
and  Winter  (2002)  defined  a  dynamic  capability  as  “a  learned  and  stable  pattern  of  collective 
activity  through  which  the  organization  systematically  generates  and  modifies  its  operating 
routines  in  pursuit  of  improved  effectiveness”  (p.  340).  Examples  of  dynamic  capabilities 
include  systematic  methods  for  changing  operating  routines  and  organizational  capabilities 
for  process  research  development,  restructuring  and  re-engineering,  and  post-firm 
acquisition  integration. 

According  to  Zollo  and  Winter  (2002),  dynamic  capabilities  are  created  through  three 
learning  mechanisms:  experience  accumulation,  knowledge  articulation,  and  knowledge 
codification,  as  shown  in  Figure  2  Knowledge  articulation  occurs  when  individuals  express 
their  opinions  and  beliefs,  challenge  each  other’s  viewpoints,  and  engage  in  constructive 
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confrontations.  Knowledge  articulation  is  a  deliberate  process  through  which  groups  and 
individuals  seek  to  understand  what  works  and  what  does  not  to  complete  a  particular 
organizational  task.  Organizational  and  individual  competence  is  enhanced  when  implicit 
knowledge  is  articulated  through  discussion,  debriefing  sessions,  and  assessments  of  past 
performance.  These  processes  serve  to  improve  individuals’  understanding  of  the  causal 
mechanisms  that  link  actions  to  outcomes.  Articulation  requires  significant  effort  but  can 
produce  improved  understanding  of  changes  in  links  between  action  and  performance. 
Articulation  can  thus  result  in  adaption  of  existing  routines. 


Learning  Mechanisms 

•  Experience  accumulation 

•  Knowledge  arr'culatbn 

•  Knowledge  codification 


Dynamic  Capabilities 

•  Process  R&D 

•  Restructuring,  re-engineering 

•  Post-acquisition  integration 


Evolution  of  Organizational  Operating  Routines 


Figure  2.  Learning,  Dynamic  Capabilities,  and  Operating  Routines 

(Zollo  &  Winter,  2002) 

Knowledge  codification,  occurs  when  articulated  understandings  are  captured  in 
writing,  as  in,  for  example,  manuals,  decision  support  systems,  or  project  management 
software.  Knowledge  codification  requires  greater  effort  than  articulation.  Codification  is 
challenging  because  it  can  be  difficult  to  ensure  that  codified  guidance  is  adequate,  and  also 
that  such  guidance  is  implemented  and  followed.  The  additional  effort  means  that 
codification  may  be  costly.  Costs  include  the  time,  resources  and  attention  invested  in  the 
development  of  task-specific  tools,  as  well  as  the  indirect  costs  of  a  possible  increase  in 
organizational  inertia  (because  the  now-codified  routine  is  applied  regularly,  making  change 
more  difficult)  or  the  inappropriate  application  of  a  codified  routine. 

The  development  of  dynamic  capabilities  is  costly.  Investments  include  financial, 
temporal,  and  cognitive  resources  that  are  directed  toward  improving  understanding  of 
action-performance  linkages.  The  level  of  investment  can  be  considered  along  a  continuum. 
It  will  be  lowest  when  a  firm  relies  on  the  accumulation  of  experience  in  a  semiautomatic 
fashion  and  more  costly  when  the  firm  relies  on  knowledge  articulation  and  even  more  so  for 
codification.  Dynamic  capabilities  require  specialized  personnel,  committed  to  change  roles, 
and,  to  be  economically  worthwhile,  an  opportunity  to  be  exercised. 

According  to  some  scholars,  organizations  operating  in  rapidly  changing  business 
environments  require  dynamic  organizational  capabilities,  which  can  be  “harnessed  to 
continuously  create,  extend,  upgrade,  protect  and  keep  relevant  the  enterprise’s  unique 
asset  base”  (Teece,  2007,  p.  1319).  However,  although  dynamic  capabilities  have  attracted 
attention,  they  are  not  the  only  means  of  organizational  adaptation  and  change.  Firms  can 
also  adapt  and  change  through  ad  hoc  or  one-time  problem  solving.  Ad  hoc  problem  solving 
is  not  repetitious  and  highly  patterned.  It  typically  occurs  in  response  to  unpredictable  events 
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in  the  environment.  Whereas  the  development  and  maintenance  of  dynamic  capabilities 
requires  ongoing  specialized  investments  in  personal  and  attention,  the  costs  of  ad  hoc 
problem  solving  disappear  when  there  is  no  problem  to  solve.  The  costs  of  ad  hoc  problem 
solving  are  largely  opportunity  costs  associated  with  the  attention  given  the  problem.  If  the 
problem  is  no  longer  presented,  attention  shifts  and  costs  are  relieved.  Thus,  so-called 
routine  capabilities,  augmented  when  needed  with  ad  hoc  problem  solving,  may  be  the  more 
cost  effective  response  to  achieving  organization  adaptation  (Winter,  2003). 

The  responses  of  the  acquisition  community  to  the  change  in  warfighters’  needs, 
exemplified  in  this  study  through  the  case  of  the  JCTD  office,  can  be  considered  a 
successful  example  of  ad  hoc  problem  solving.  The  reaction  of  the  community;  creating 
organizations  and  processes  to  fill  a  particular  need  from  existing  organizations,  budgets, 
and  processes;  learning  on  the  job;  and  forging  one-time  solutions  are  all  examples  of  ad 
hoc  problem  solving,  creative  innovation  to  a  particular  problem. 

As  discussed  previously,  such  problem  solving  may  be  more  cost  effective  than 
creating  a  dynamic  capability.  This  is  particularly  true  when  an  environment  is  ambiguous 
and  unpredictable  or  competitors  are  likely  to  copy  one’s  success.  The  long-term  response 
to  the  need  for  rapid  fielding  during  the  conflicts  in  Iraq  and  Afghanistan  should  take  into 
account  the  “success”  of  this  problem-solving  approach.  An  evolutionary  approach  to 
organizational  change  would  suggest  that  the  variation  of  organizations  and  processes  be 
subject  to  environmental  selection,  whereby  only  those  exhibiting  fit  with  the  environment 
are  likely  to  survive.  Thus,  if  in  fact  rapid  fielding  remains  a  paramount  need,  we  would 
expect  the  creativity  that  fostered  the  organizations  that  met  that  need  to  find  a  way  to 
continue  to  meet  it.  History  suggests  that  those  within  the  DoD  are  adept  at  doing  this. 
Alternatively,  however,  if  rapid  fielding  is  not  required,  the  costs  of  developing  this  “dynamic 
capability”  may  be  misplaced. 

DoD  acquisition  has  exhibited  a  long  history  of  resistance  to  change.  Given  the 
bureaucratic  make-up  of  the  DoD  and  the  size  of  the  organization,  this  is  not  surprising. 
Further,  bureaucratic  processes  are  appropriate  in  some  situations  (particularly  those 
involving  great  risk)  and  may  be  a  necessity  for  the  DoD.  However,  as  many  have  noted, 
DoD  organization  structure  and  processes  were  well  adapted  to  the  post-WWII-Cold  War 
era,  and  since  2001  that  stable  environment  no  longer  exists.  Thus,  many  DoD  routine 
capabilities  may  have  technical  fit — they  fit  well  with  a  particular  function,  such  as  the 
acquisition  of  large,  complicated  weapons  systems  to  meet  the  needs  of  many  players  when 
time  and  money  are  abundant — but  may  not  fit  with  the  new  environment.  The  question  then 
becomes,  what  is  the  best  way  to  adapt  to  the  new  environment. 

One  must  be  somewhat  cautious  in  making  direct  comparisons  between  the 
competitive  business  environment — where  success  is  generally  defined  as  earning  greater 
financial  returns  than  one’s  rival — and  the  multifaceted  environment  facing  the  DoD 
acquisition  community.  The  discussion  above  suggests  that  ad  hoc  problem  solving  should 
not  be  discounted  out  of  hand  and  without  consideration.  Such  solutions  allow  the  DoD  to 
adapt  in  low  cost  manner  without  attempting  to  change  the  overall  bureaucracy.  Although 
developing  dynamic  organizational  capabilities  may  be  possible,  doing  so  is  clearly  costly 
and  difficult,  as  exemplified  by  the  many  failed  attempts  with  the  DoD  and  in  industry.  An 
alternate  perspective  on  ad  hoc  problem  solving  suggests  that  these  solutions  should  be 
rewarded,  and  perhaps  structural  changes  should  be  designed  to  allow  such  solutions  to 
emerge  and  dissipate  as  needed,  rather  than  automatically  seeking  codification, 
centralization,  and  formalization.  This  is  particularly  salient  if  one  considers  that  the 
environment  may  continue  to  change.  The  organizations  and  processes  that  have  emerged 
and  evolved  to  exhibit  technical  and  environmental  fit  for  the  environment  following  the 
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September  2001  attacks  may  not  fit  the  environment  of  the  future.  Ad  hoc  problem  solving  is 
a  low-cost  alternative  for  allowing  adaptability  within  the  large  bureaucracy. 

Implementing  Change — An  Additional  Consideration 

As  noted  above,  this  research  suggests  that  reforms  should  consider  how  to  take 
advantage  of  the  ad  hoc  problem  solving  skills  of  the  DoD  acquisition  community. 
Furthermore,  the  discussion  suggests  that,  when  codification  of  learning  is  undertaken,  then 
much  of  the  value  of  such  efforts  lies  in  the  process,  rather  than  in  the  end.  Capturing  this 
value  requires  a  collaborative,  “safe”  environment  that  facilitates  knowledge  sharing.  The 
acquisition  community  can  be  viewed  as  a  system,  composed  of  many  different  types  of 
actors  and  organizations,  operating  in  an  uncertain  environment  subject  to  shocks  and 
subsequent  turbulence.  Although,  some  competition  within  systems  is  beneficial,  a  long 
history  of  research  documents  the  deleterious  effects  of  competitive  environments  on 
knowledge  sharing  at  the  individual  level  and  of  prices  wars  and  “hyper-competition”  on 
industry  profitability  at  the  systems  level.  Policymakers  should  be  aware  of  the  potential 
consequences  of  such  negative  competition  and  structure  reforms  to  minimize  its  likelihood. 

Scholars  argue  that  in  business  landscapes  characterized  by  great  turbulence, 
traditional  competitive  actions  may  not  lead  to  an  advantage  but  may  rather  result  in  further 
turbulence.  For  example,  organizations  relying  on  dynamic  capabilities  to  “turn  themselves 
into  moving  targets”  moving  faster,  changing  more  quickly  to  avoid  being  “leapfrogged”  by 
competitors,  may  increase  field  level  turbulence  (Delapierre  &  Mytelka,  1998,  p.  78;  Selksky 
et  al.  2007,  p.  79).  Selksy  et  al.  (2007)  argued  that  success  in  turbulent  environments  hinges 
on  collaborative  endeavors  to  develop  new  field  level  processes,  adaptive  skills,  and 
capabilities.  Selsky  et  al.  (2007)  illustrated  these  dynamics  referencing  a  pair  of  studies  of 
hospitals  in  hyper-turbulent  environments.  In  response  to  changes  in  federal  Medicare 
reimbursement  programs,  the  states  of  California  and  Minnesota  each  made  major  reforms 
to  their  healthcare  systems,  resulting  in  a  turbulent  business  environment.  However,  the 
healthcare  industries  in  the  two  states  experienced  different  outcomes. 

In  1982,  California  adopted  a  managed  competition  program  in  healthcare,  creating 
incentives  for  providers  to  compete  on  price  for  government  care  for  indigent  citizens.  At  the 
same  time,  the  federal  government  changed  Medicare  reimbursement  procedures. 

Together,  these  events  resulted  in  unanticipated  turbulence  in  the  business  landscape  of  the 
state’s  hospitals. 

California’s  hospitals  reacted  immediately,  over  one  six-week  period  during  the 
study,  two  hospitals  merged,  one  was  acquired,  and  seven  out  of  30  hospitals  experienced 
CEO  succession.  The  hospitals  entered  mergers,  alliances,  and  partnerships  between 
hospitals,  physicians,  and  insurance  plans.  These  actions  challenged  traditional  rules  of 
competition  within  the  industry,  understandings  about  the  domain  and  identity  of  hospitals, 
and  the  traditional  boundaries  between  players  in  the  healthcare  field.  For  example,  insurers 
became  deliverers  of  care  through  investments  in  managed  care  organizations,  hospitals 
became  providers  of  care  through  offsite  clinics,  invading  the  traditional  domain  of  doctors, 
and  physicians  took  on  new  risks  for  the  cost  and  quality  of  the  services  they  offered  by 
signing  preferred  or  exclusive  provide  contracts. 

In  response,  the  hospitals  formed  integrated  networks  seeking  access  to  new 
markets,  economies  of  scope  and  scale,  and  complements  to  their  distinctive  competencies. 
However,  as  the  environmental  turbulence  continued  to  increase,  the  hospitals  reacted  with 
hyper-competitive  moves  actively  disrupting  previous  competitive  norms  and  each  other’s 
competitive  advantages.  For  example,  preferred  provider  networks  linked  groups  of 
physicians  to  particular  provider  hospitals  and  health  plans.  This  restricted  other  hospitals’ 
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access  to  these  physicians  and  spawned  a  bidding  war.  Medical  staffs  that  had  taken 
hospitals  years  to  develop  were  decimated.  Overtime,  the  competitive  actions  ceased  to 
provide  advantage  and  success  and  became  only  a  requirement  for  survival.  Smaller 
players  were  marginalized  as  larger,  stronger  organizations  consolidated  their  control  over 
resources.  The  region’s  healthcare  system  continues  to  suffer  from  “huge  systemic  flaws: 
Rampant  inflation,  large  numbers  of  uninsured,  uneven  and  hard  to  measure  quality  and 
uncertain  funding”  (Rauber,  2005;  Selsky,  2007). 

In  response  to  the  federal  changes,  Minnesota  reconfigured  its  healthcare  industry  a 
decade  later.  Healthcare  providers  responded  initially  in  a  manner  similar  to  those  in 
California.  However,  in  contrast  to  California’s  hospital  executives,  those  in  Minnesota 
viewed  themselves  as  the  architects  of  a  new  organizational  model.  Minnesota’s  executives 
constructed  collaborative  networks  yielding  “win”  solutions  for  many  players  in  the  field. 
While  vigorous  competition  continued,  executives  were  able  to  anticipate  some  of  the 
negative  effects  of  their  individual  competitive  actions  in  the  extended  field  and  to  create  a 
model  of  competition  that  partially  controlled  for  those  effects. 

In  the  end,  the  process  of  industry  restructuring  in  California  generated  negative 
externalities,  whereas  industry  transformation  in  Minnesota  retained  negative  feedback 
brakes  and  avoided  some  of  these  effects.  As  illustrated  by  these  examples,  hyper¬ 
competition  in  a  turbulent  environment  can  result  in  unanticipated  negative  effects.  In 
California,  failures  to  develop  sustainable,  collective  strategies  “echo  in  the  form  of  failed 
alliances,  labor  problems  and  uncertain  financial  health”  (Selsky,  2007),  whereas  the 
collaborative  efforts  of  hospitals  in  Minnesota  contributed  to  a  more  successful,  field-level 
change. 

If  successful  adaptation  in  a  turbulent  environment  is  best  achieved  through 
collaborative  effort,  it  is  imperative  that  such  collaboration  between  field  players  be  fostered. 
Although  comparisons  between  a  competitive  business  environment  and  a  public  agency 
are  not  absolute,  they  can  be  enlightening.  In  the  field  of  defense  acquisition,  there  are 
many  players.  As  in  the  hospital  examples  above,  an  environmental  change  resulted  in  a 
redefinition  of  the  domain  and  roles,  the  emergence  of  new  entities  and  partnerships,  and 
the  creation  of  new  processes.  If  changes  to  the  system  lead  to  “hyper-competitive” 
behavior  among  the  new  players  in  the  acquisition  field  now  facing  restructuring  and/or 
between  the  new  and  traditional  players,  unanticipated  negative  outcomes  can  be  expected. 

This  suggests  that  if  substantial  reorganization  and  or  codification  of  emergent 
processes  is  undertaken,  the  DoD  should  consider  how  to  foster  collaboration  between  the 
newly  formed  organizations  to  develop  roles  and  patterns  of  interaction  viewed  as  “wins”  for 
multiple  players  in  the  field.  Structural  reform  should  be  complemented  by  efforts  to  solicit 
and  incorporate  inputs  from  new  and  traditional  field  players  with  a  view  toward  crafting  a 
field  solution.  Achieving  “the  hope  that,  over  time,  the  DoD  acquisition  community  will 
understand  the  benefits  of  the  rapid  approach — and  the  countercultural  stigma  will  dissolve” 
(Gansler,  2009,  p.  26)  may  require  active  intervention  to  change  perceptions,  and  at  the 
very  least,  a  thoughtful  consideration  of  how  to  avoid  worsening  the  problem  when  making 
structural  changes. 

Conclusion 

This  report  describes  the  preliminary  analysis  and  findings  of  our  study  exploring 
what  drives  successful  organizational  adaptation  in  the  context  of  technology  transition  and 
acquisition  within  the  DoD.  It  is  based  on  our  initial  collection  and  analysis  of  archival  and 
interview  data.  Our  preliminary  analysis  suggests  that  ad  hoc  problem  solving  may  be  an 
undervalued  yet  broadly  practiced  skill  set  within  the  DoD.  We  are  currently  conducting  a 
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second  round  of  targeted  interviews  designed  to  illuminate  how  those  in  the  JCTD  office 
used  ad  hoc  problem  solving  and  organizational  routines  to  field  technology  solutions.  We 
will  use  the  data  to  further  explore  how  ad  hoc  problem  solving  may  be  used  to  support 
adaptive  responses  within  the  DoD  acquisition  community  and  to  explicate  criteria  for 
determining  when  to  rely  on  ad  hoc  problem  solving  versus  when  to  invest  in  creating 
dynamic  organizational  capabilities. 
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Abstract 

This  report  presents  the  results  of  research  and  analyses  on  current  and  future  operational 
capability  gap  development  and  acquisition  practices  in  the  United  States  Navy  and  the 
Combatant  Commands  (COCOMs),  as  exemplified  by  Pacific  Command  (PACOM). 
Leveraging  key  stakeholder  interviews  and  using  a  systems  thinking  framework  known  as  the 
Conceptagon  (Boardman  &  Sauser,  2008),  we  investigated  and  assessed  the  Navy’s  Future 
Naval  Capabilities  (FNC)  process  as  well  as  the  Joint  Staff  (JS)  Capability  Gap  Assessment 
(CGA)  process  as  it  applies  to  the  annual  submission  of  PACOM’s  Integrated  Priority  List 
(IPL)  of  capability  needs.  The  study  approached  both  processes  as  systems  and  identified 
and  explored  their  critical  systemic  attributes  such  as  parts,  relationships,  boundaries, 
governance  mechanisms/structures,  key  processes,  transformations,  stakeholders,  and 
missions,  to  name  a  few.  Based  on  this  assessment,  we  conducted  a  structured  and 
systematic  comparison  of  the  two  processes  to  identify  good  practices  and  favorable 
dynamics  that  are  likely  to  reinforce  the  desired  outcome,  which — for  our  purposes — is 
defined  as  the  resolution  of  capability  gaps  and,  ultimately,  deployment  of  needed  capabilities 
to  the  warfighters.  In  light  of  this  analysis,  we  present  key  insights,  explore  some  problem 
areas,  and  discuss  possible  improvements  to  the  said  processes. 

Summary  of  Results 

The  following  bulleted  list  presents  the  study  team’s  conclusions  and  key  insights,  and  it 
is  followed  by  a  list  of  key  preliminary  considerations  for  improvement  strategies. 
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Key  Insights  Into  Effective  Practices  of  the  FNC  Program 

•  Participatory  and  binding  measures  (as  exemplified  by  FNC  roundtable  and 
technology  transition  agreements)  create  a  sense  of  collective  process 
ownership  amongst  stakeholders  who  may  otherwise  have  differing  interests. 

This  increases  the  chance  of  solution  resourcing  and  development. 

•  Communicating  intended  outputs  and  outcomes  relative  to  the  gap  development 
process  (as  done  by  FNC  in  road  shows  and  associated  briefing  materials)  helps 
to  manage  stakeholder  expectations  and  inform  stakeholder  perceptions  of 
success,  culminating  in  improved  stakeholder  commitment  and  acceptance  of 
gap  development  processes  and  associated  fulfillment  activities. 

•  FNC  identification  and  tracking  of  gap  fulfillment  measures  (e.g.,  transition 
statistics)  allows  the  FNC  program  to  adjust  in  order  to  improve  performance. 

•  FNC  integration  of  processes  that  represent  the  entire  lifecycle  of  a  gap  (from 
gap  identification  through  capability  deployment  to  the  warfighter)  promotes  a 
seamless  transition  between  different  phases  of  the  effort,  facilitates  flow  of 
required  information,  and  provides  for  continuity  of  efforts. 

How  FNC  Practices  Could  Improve  the  JS  CGA/PACOM IPL  Process 

•  The  JS  CGA/PACOM  IPL  process  could  expand  its  boundary  to  include  solution 
providers.  Existing  structures  (e.g.,  Functional  Capability  Boards  [FCBs], 
supporting  working  groups)  could  be  used  to  facilitate  formal  participation  by  the 
acquisition  community. 

•  COCOM  organizations  could  receive  feedback  on  JS  disposition  of  gap 
information.  Formal,  accountable,  ongoing  and  two-way  communication  would 
facilitate  understanding,  expectation  management,  and  feedback.  The  JS  could 
establish  formal  communication  mechanisms  to  provide  updates  on  gap 
modifications  (i.e.,  merging  similar  gaps,  capability  board  adjustments)  and 
outcomes. 

•  Gap  and  solution  progress  statistics,  collected  in  coordination  with  the  acquisition 
and  warfighter  communities  (currently  partially  tracked  by  the  JS  and  accessible 
through  Knowledge  Management  and  Decision  Support  [KMDS]),  could  be 
documented  and  published  periodically  (e.g.,  annually  or  bi-annually)  and  could 
be  actively  disseminated  to  COCOMs  and  briefed  at  FCBs  and  Joint  Capability 
Boards  (JCBs). 

•  These  outcome-tracking  metrics  could  be  used  by  both  JS  and  COCOM  staffs  to 
inform  process  improvement  efforts.  The  JS  process  could  establish  procedures 
and  update  instructions  and  guidance  documents  to  better  define  roles  and 
responsibilities  with  metric  and  process  accountability. 

Introduction 

The  Department  of  Defense’s  (DoD)  annual  Integrated  Priority  List  (IPL)  process1  is 
an  integral  part  of  the  Joint  Staff’s  (JS)  Capability  Gap  Assessment  (CGA)  process,2  which, 
once  published,  is  one  input  among  many  considered  by  defense  acquisition  communities. 
This  process,  however,  may  result  in  a  high-priority  IPL  capability  gap(s)  receiving  lower 
priority  consideration  in  the  military  departments’  resource  allocation  and  acquisition 
decisions.  By  contrast,  the  Navy’s  Future  Naval  Capabilities  (FNC)  process  is  considered  by 
both  naval  and  non-naval  audiences  to  be  a  remarkably  responsive  and  integrated  process 

1  The  IPL  process  is  intended  to  address  the  nine  Combatant  Commands’  Joint  warfighting  needs 
through  identification  and  prioritization  of  operational  capability  gaps. 

2  Results  of  the  CGA  process  are  published  through  a  Joint  Requirements  Oversight  Council 
Memorandum  (JROCM). 
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that  supports  naval  warfighter  needs.  This  is  further  substantiated  by  the  FNC  process 
transition  statistics,  which  demonstrate  a  record  of  successfully  addressing  naval  capability 
gaps  through  the  allocation  of  resources  and  establishment  of  effective  schedules  for 
research,  development,  and  acquisition  of  needed  capabilities  (Office  of  Naval  Research, 
2012). 

In  this  paper,  we  present  a  comparative  assessment  of  the  Navy’s  FNC  process  and 
the  JS  CGA/COCOM  IPL  process  as  exemplified  in  the  PACOM  IPL  process.  This  study 
acknowledges  FNC  is  an  end-to-end  process  inclusive  of  solution  development  and 
transition,  whereas  the  JS  CGA/COCOM  IPL  process  focuses  solely  on  gap  identification 
and  assessment  with  little  integration  of  solution  activities.  Although  this  comparison  may 
first  appear  asymmetric,  we  compared  the  processes  across  similar  steps  that  culminate  in 
the  shared  milestone  of  establishing  final,  prioritized  gaps  (see  Figure  1).  Throughout  the 
course  of  the  study,  however,  it  was  frequently  noted  that  integrated  solution  development 
activities  significantly  influenced  FNC’s  gap  development  activities.  As  will  be  explained  in 
our  conclusions,  in  FNC,  the  interaction  and  mutual  feedback  between  these  steps 
maximizes  the  process’  overall  responsiveness  to  naval  warfighters’  needs. 
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Figure  1.  Common  Milestones  of  FNC  and  JS  CGA/IPL 

Methodology 

A  Systems  Thinking  Approach  to  Data  Compilation  &  Comparison 

The  team  chose  to  approach  data  analysis  using  a  systems  thinking  approach,  which 
is  particularly  useful  when  assessing  soft  systems,  such  as  enterprises  and  processes.  In 
this  study,  the  FNC  process  and  the  JS  CGA/PACOM  IPL  development  process  were 
viewed  as  systems,  each  with  constituent  parts,  united  by  relationships,  working  together  to 
achieve  a  specific  purpose. 

In  particular,  the  team  used  the  soft  systems  framework,  the  Conceptagon 
(Boardman  &  Sauser,  2008),  to  assess  the  FNC  process  and  the  JS  CGA /  PACOM  IPL 
process.  The  Conceptagon  served  as  a  consistent  framework  for  comparison,  standardizing 
the  team’s  approach  to  characterizations.  It  allowed  the  team  to  compare  seemingly 
disparate  processes  across  set  dimensions  of  interest.  For  example,  instead  of  trying  to 
compare  the  Joint  Staff  J-8  with  the  FNC  IPT  (a  difficult  comparison  at  its  face  value),  the 
team  compared  across  more  abstract  dimensions  such  as  “actors”  and  “relationships.”  Such 
dimensions  of  comparison  were  used  throughout  the  study  to  facilitate  a  consistent  and 
coherent  approach  to  identifying  the  underlying  similarities  and  differences  between  the  two 
processes. 

Data  Collection 

Data  and  relevant  information  for  this  study  were  collected  through  literature  review 
and  interviews.  To  prepare  for  assessment  of  FNC,  study  team  members  attended  the 
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Office  of  Naval  Research’s  FNC  external  training  (Office  of  Naval  Research,  2012).  This 
half-day  course  provided  instruction  on  FNC  program  basics  (FNC  goals,  objectives,  and 
participants)  and  key  program  management  processes.  In  addition  to  the  half-day  course, 
the  team  conducted  interviews  with  the  FNC  Program  Director  (Steve  Smolinski)  and 
members  of  the  director’s  staff.  The  team  also  reviewed  Integrated  Process  Team  (IPT) 
charters,  FNC  briefings,  FNC  policy  memorandums,  and  related  naval  instructions.  To 
prepare  for  assessment  of  PACOM’s  IPL,  and  its  movement  through  the  broader  Joint  Staff 
CGA  process,  the  team  reviewed  PACOM  documents  such  as  Plans  to  Resources  to 
Outcomes  Process  (PROP)  briefings.  The  team  also  conducted  interviews  with  participants 
in  the  PACOM  IPL  process  such  as  the  PACOM  science  advisor  and  PROP  and  future 
capabilities  subject-matter  experts.  To  understand  what  happens  once  IPL  entries  are 
submitted  to  the  Joint  Staff,  the  team  interviewed  members  of  the  Joint  Staff  J-8,  Joint 
Capability  Office.  The  team  also  reviewed  a  number  of  applicable  Chairman  of  the  Joint 
Chiefs  of  Staff  Instructions  (CJCSIs).  For  general  awareness,  cleared  members  of  the  team 
reviewed  additional  information,  including  actual  IPL  entries. 

Research  Questions  and  Study  Scope  Adjustments 

This  study’s  original  research  question  sought  to  address  whether  the  acquisition 
process  is  responsive  to  COCOM  needs  by  comparing  a  very  responsive  system  (the 
Navy’s  FNC)  to  the  existing  JS/COCOM  system,  using  PACOM  as  an  example.  To  answer 
this  question,  the  team  needed  to  explore  four  major  research  areas:  (1)  how  COCOM 
needs  are  identified  (collection  of  warfighter  statements  of  operational  shortfalls),  (2)  how 
operational  shortfalls  are  captured  and  communicated  (development  of  the  COCOM  IPL), 

(3)  how  IPL  statements  are  transformed  into  Joint  Capability  Gap  statements  (development 
of  the  Joint  Capability  Gap  Assessment),  and  (4)  how  Joint  Capability  Gaps  are  resolved 
(Service  efforts  to  fund  and  develop  solutions).  While  the  fourth  research  area  may  hold 
specific  statistics  that  seemingly  answer  the  question,  these  statistics  are  subject  to  great 
error  if  the  gaps  to  which  the  Services  are  responding,  do  not  accurately  reflect  COCOM 
needs.  We  realized  that  answering  the  question  was  not  just  a  matter  of  gathering  statistics 
on  Service  development  programs,  but  also  of  exploring  the  gap  development  process  that 
informs  gap  resolution  efforts.  Therefore,  we  focused  on  the  gap  development  processes 
(Areas  1-3)  rather  than  Service  solution  funding  and  development  (Area  4),  which  would 
have  been  a  study  in  its  own  right. 

In  addition  to  focusing  on  the  gap  development  process,  the  study  determined 
focusing  on  a  single  COCOM  would  yield  a  more  detailed  analysis.  The  decision  to  examine 
PACOM  was  made  for  several  reasons:  (1)  PACOM  had  a  mature  gap  identification 
process;  (2)  PACOM’s  process  is  well  documented  and  the  team  had  access  to  PACOM 
briefings,  instructions,  and  materials;  and  (3)  the  team  had  personal  contacts  in  the  PACOM 
IPL  development  process  who  agreed  to  participate  in  interviews. 

Limitations  of  the  Study 

Due  to  the  decision  to  focus  on  a  single  COCOM,  the  resulting  recommendations 
may  be  less  applicable  to  some  COCOMs.  Also,  information  used  throughout  the  study  was 
based  on  access  to  available  documentation  and  the  professional  views/perceptions  of  the 
individuals  interviewed;  as  a  result,  some  of  the  information  is  subject  to  personal  bias  and 
limited  to  the  experience  of  those  interviewed.  Finally,  to  remain  unclassified,  specific  IPL 
information  is  not  discussed. 
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Overview  of  the  FNC  and  Joint  Staff  CGA/IPL  Systems 

The  following  section  presents  a  high-level  overview  of  both  FNC  and  the  JS 
CGA/PACOM  IPL  processes  and  provides  grounds  for  understanding  the  subsequent 
Conceptagon  assessment. 

Future  Naval  Capabilities 

Initiated  in  2002  (Office  of  Naval  Research,  n.d.-b),  the  FNC  program  and  the 
associated  process  addresses  naval  gaps  with  science  and  technology  (S&T)  solutions  on 
an  annual  basis  (Office  of  Naval  Research,  2012).  Using  6.2  (i.e.,  applied  research)  and  6.3 
(i.e. ,  advanced  technology  development)  funding,  this  program  “develop[s]  ...  quantifiable 
technology  products  in  response  to  validated  S&T  gaps”  within  a  five-year  time  frame  (Office 
of  Naval  Research,  2012).  Upon  maturation  of  technology  and  fulfillment  of  exit  criteria 
(Office  of  Naval  Research,  2012),  the  FNC  program  transitions  related  products  to  naval 
acquisition  programs  of  record  “for  timely  incorporation  into  platforms,  weapons,  sensors, 
and  process  improvements”  (U.S.  Naval  Research  Laboratory,  n.d.).3  The  FNC  process  is 
currently  organized  along  “9  pillars  of  Enabling  Capabilities  (ECs)”  (Office  of  Naval 
Research,  2012),  each  of  which  “is  an  aggregate  of  science  and  technology  that  is  aligned 
to  an  identified  warfighting  gap  or  warfighting  capability,  and  it  can  deliver  a  distinct, 
measurable  improvement  that  contributes  to  closing  the  corresponding  warfighting  gap” 
(Office  of  Naval  Research,  n.d. -a). 4 

The  FNC  program  is  managed  through  a  collaborative  process.  Broadly  speaking,5 
OPNAV/HQMC  requirements  are  assessed  to  identify  gaps  with  S&T  solutions.  These  gaps 
are  then  assigned  to  related  pillars  for  identification  of  potential  solutions.  Integrated  Product 
Teams  (IPTs)  forward  prioritized  ECs.  These  ECs  are  reviewed,  assessed,  and  approved  by 
the  Technical  Oversight  Group  (TOG),  a  three-star  Navy  and  Marine  Corps  Board  of 
Directors.  Related  products  begin  the  development  phase  with  strict  conditions  for  an 
eventual  transition  to  the  warfighter  as  agreed  amongst  representatives  from 
requirements/resource  communities,  science  and  technology  developers,  and  the 
acquisition  community.  The  overall  program  is  administered  by  the  Office  of  Naval  Research 
(ONR). 

The  DoD  Joint  Staff  Capability  Gap  Assessment/Integrated  Priority  List  Process  and 
Its  Employment  by  Pacific  Command 

The  annual  CGA/IPL  process  produces  a  prioritized  list  of  DoD  warfighting  capability 
gaps  that  impact  DoD  Combatant  Commands’  (COCOMs)  execution  of  operational, 
contingency,  and  campaign  plans.  This  process  informs  the  JS  CGA,  Functional  Capability 
Board  (FCB)  planning  guidance,  and  development  of  the  Chairman’s  Program  Assessment 
(CPA),  and  analyzes  baseline  resource  allocation  priorities  for  the  subsequent  years’  IPLs 
(K.  Carlan,  personal  communication,  April  11, 2012;  D.  Glenister,  personal  communication, 
April  16,  2012). 


3  The  FNC  program  deals  only  with  products  whose  technology  readiness  levels  (TRL)  are  between 
three  and  six  (Office  of  Naval  Research,  2012). 

4  As  explained  in  the  FNC  external  training  (ONR,  2012),  EC  pillars  include  sea  shield,  sea  strike,  sea 
basing,  FORCEnet,  naval  expeditionary  maneuver  warfare,  capable  manpower,  force  health 
protection,  enterprise  and  platform  enablers,  and  power  &  energy. 

5  The  discussion  of  the  FNC  process  relies  on  information  gathered  during  FNC  Training  attended  at 
the  ONR  on  March  9,  2012,  and  the  PowerPoint  slides  (ONR,  2012)  disseminated  during  the  training. 
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Each  COCOM  has  a  different  process  for  generating  input  into  the  DoD  Joint  Staff 
CGA/I PL6  process.  PACOM’s  employment  of  the  JS  CGA/IPL  process  is  comprised  of  four 
fundamental  activities,  including  development,  submission,  assessment,  and  validation  of 
capability  gaps  (K.  Carlan,  personal  communication,  April  11, 2012;  Carlan,  201 1).7  First, 
the  process  begins  with  the  identification,  organization,  and  development  by  PACOM 
components  and  sub-Unified  Command  organizations  of  capability  gaps  through  a 
collaboratively  facilitated  Plans  to  Resources  to  Outcomes  Process  (PROP)  within  the 
PACOM  J-8  Resources  and  Assessment  Directorate.  USF-J,  USFK,  USARPAC,  PACFLT, 
MARFORPAC,  PACAF,  SOCPAC,  JIATF-West,  JPAC,  and  ALCOM  are  the  participating 
PACOM  PROP  organizations.  The  PROP  results,  used  for  a  number  of  PACOM  purposes, 
including  as  an  input  to  the  I  PL,  are  reviewed  and  revised  by  key  0-6  level  staff  officers,  J- 
code  Directors,  and  the  PACOM  Deputy  Commander.  Second,  the  PACOM  Commander 
then  prioritizes,  approves,  and  submits  the  PACOM  IPL  to  the  Joint  Staff  in  support  of  the 
JS  J-8  Force  Structure,  Resources,  and  Assessment  Directorate  CGA  process,  and 
informally  to  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics; 
USD[AT&L]).  Third,  JS  J-8  conducts  a  CGA  of  all  nine  COCOM  IPLs  including  PACOM  (K. 
Duffy  &  D.  Glenister,  personal  communication,  April  16,  201 2). 8  The  assessment  process 
includes  review  and  analysis  of  IPL  capability  gaps  by  nine  FCBs,  which  determine 
warfighting  relevance  and  result  in  recommended  capability  solutions  and  funding.  This 
process  produces  and  submits  a  J-8-recommended  prioritized  list  of  capability  gaps  and 
associated  solutions  to  the  Joint  Capability  Board  (JCB).  The  JCB  reviews  and 
recommends  the  capability  gap  list  to  the  Joint  Requirements  Oversight  Council  (JROC). 
Fourth,  the  JROC  validates  capability  gaps,  and  publishes  a  JROC  Memorandum  (JROCM) 
CGA,  which  is  distributed  to  the  USD(AT&L)  and  Services  for  action  on  capability  solutions 
acquisition  and  fielding. 

The  Conceptagon  Assessment 

The  Conceptagon  framework  (Boardman  &  Sauser,  2008)  aids  analysts  in 
conceptualizing  and  characterizing  a  system.  The  framework  defines  a  system’s  attributes 
in  seven  easy-to-remember  sets  (triplets).  This  analysis  used  the  Conceptagon  triplets.  Our 
understanding  of  these  triplets  is  explained  in  the  following  list: 

•  Interior,  exterior,  boundary — This  triplet  describes  the  perimeter  that 
separates  entities  that  comprise  the  system,  from  entities  outside  the 
system’s  control. 

•  Wholes,  parts,  relationships — This  triplet  requires  identification  of  the  system 
at  hand,  the  constituent  pieces,  and  the  relationships  that  bind  those  pieces 
together. 

•  Structure,  function,  process — This  triplet  identifies  the  key  composition, 
arrangement,  or  organization  (structures)  a  system  employs  to  support  key 
activities  (processes)  necessary  to  produce  the  desired  system  behavior 
(function). 


6  In  some  cases,  the  study  team  refers  to  the  JS  CGA/IPL  process  as  opposed  to  the  JS 
CGA/PACOM  IPL  process.  This  change  in  nomenclature  is  intended  to  capture  instances  wherein 
discussion  points  are  likely  relevant  to  multiple  COCOMs. 

7  All  discussion  on  PACOM  IPL  activities,  including  PROP,  is  based  on  personal  interviews  with  Kit 
Carlan  and  Ken  Bruner  of  PACOM,  and  PowerPoint  slides  dated  March  23,  2011,  and  prepared  by 
Kit  Carlan. 

8 The  information  about  JS  CGA  process  and  details  is  based  on  interviews  with  Air  Force  Col  Keith 
Duffy  of  JS  and  Navy  CAPT  Dave  Glenister  of  JS. 
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•  Inputs,  outputs,  transformations — This  triplet  identifies  items  coming  into  the 
system  (inputs)  and  items  exiting  the  system  (outputs)  as  products  or 
deliverables.  The  triplet  also  identifies  the  change  (i.e.,  transformation)  that 
converts  inputs  to  outputs. 

•  Command,  control,  communication — This  triplet  explores  the  system’s 
governance  structures  and  control  mechanism,  and  takes  into  account 
communication  feedbacks  and  stovepipes. 

•  Openness,  hierarchy,  emergence — This  triplet  investigates  a  system’s  ability 
to  accept  inputs  from  the  exterior  environment  and  to  absorb  and 
accommodate  new  components  (openness),  to  reconfigure  itself  in  light  of 
new  requirements  (hierarchy),  and  to  respond  to  and  manage  unexpected 
behaviors  produced  by  such  changes  (emergence). 

•  Variety,  harmony,  parsimony — This  triplet  refers  to  the  system’s  design 
balance,  assessing  if  the  system  has  “too  much”  or  “too  little”  of  anything. 

Triplet  1:  Interior,  Exterior  Boundary 

Navy’s  FNC  Process 

There  are  three  pertinent  FNC  boundaries:  program  specification,  temporal,  and 
stakeholder.  The  first  boundary  is  established  in  reference  to  program  specifications,  which 
provide  that  the  FNC  system  works  only  with  6.2  and  6.3  funding  and  is  only  authorized  to 
handle  naval  gaps  with  S&T  solutions  of  Technology  Readiness  Level  (TRL)  3  to  6  (Office  of 
Naval  Research,  2012).  The  second  boundary  defines  what  is  within  and  beyond  the  system 
from  a  temporal  angle  as  portrayed  by  Figure  2.  Accordingly,  operational  requirements 
definition  takes  place  prior  to  the  FNC  process.  As  explained  in  the  FNC  training  provided 
on  March  9,  2012,  the  process  starts  with  development  of  S&T  gaps  by  the  Chief  of  Naval 
Operations  and  Fleadquarters,  Marine  Corps  staff  and  continues  as  follows:  ONR  responds 
to  the  gaps  by  proposing  ECs,  an  aggregate  of  one  or  more  technology  products  aimed  at 
closing  or  mitigating  these  gaps  (S.  Smolinksi,  personal  communication,  November  29, 
2012).  Transition  of  matured  S&T  products  into  the  acquisition  POR  is  also  within  the 
bounds  of  the  FNC  program.  Integration  to  the  acquisition  POR,  however,  happens 
subsequent  to  the  FNC  process  (Office  of  Naval  Research,  2012).  Finally,  the  third 
boundary  defines  stakeholders  who  have  influence  on  the  FNC  system.  For  example,  those 
stakeholders  that  actively  take  part  within  the  FNC  process  involve  IPTs,  the  TOG,  TOG 
working  groups  (WGs),  resource  sponsors,  acquisition  sponsors,  and  S&T  developers. 

Some  of  the  passive  FNC  stakeholders  include  OPNAV,  FCC/MCCDC,  naval  warfighters, 
COCOMs,  and  industry. 


Figure  2.  FNC  Temporal  Boundaries 
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The  FNC  boundaries  can  be  characterized  as  semi-porous,  presenting  degrees  of 
openness  across  elements,  actors,  and  issues.  Acquisition  sponsors,  resource  sponsors, 
and  S&T  developers  leverage  system  flexibility  to  reach  out  to  and  bring  in  actors  from  the 
external  environment.  This  is  not  to  say  that  the  FNC  boundaries  allow  for  complete 
permeability.  The  program  specifications  are  clear  and  firm  (e.g.,  non  S&T  gaps  are  outside 
the  authorization,  budget  has  a  clear  mandate)  and  set  structures  with  clear  membership 
descriptions  indicate  that  membership  is  not  ad  hoc  and  does  not  change  over  time.  The 
FNC  process  has  well-defined  boundaries. 

Joint  Staff  CGA  /PACOM I  PL  Process 

When  we  consider  what  is  within  and  beyond  the  JS  CGA /  PACOM  I  PL  process,  two 
boundaries  appear  particularly  relevant:  conceptual/temporal  and  stakeholder.  According  to 
the  conceptual/temporal  boundaries  (Figure  3),  PACOM  warfighters  provide  requirements 
into  the  PROP  process,  which  generates  PACOM  capability  gaps/shortfalls  for  the  PACOM 
Deputy  Commander’s  review  and  assessment  (K.  Carlan,  personal  communication, 
December  10,  2012).  The  PACOM  Commander  approves  and  submits  the  final  PACOM  IPL 
to  the  JS  CGA  process.  After  going  through  several  steps  within  this  process,  a  JROCM  is 
issued  and  conveyed  to  the  COCOMs,  Services,  and  OSD  offices.  The  second  boundary 
applicable  to  the  PACOM  IPL  process  is  the  stakeholder  boundary.  Active  stakeholders 
include  the  PACOM  Commander,  PACOM  components  and  J-code  Directors,  JS  Functional 
Capability  Board,  JS  Joint  Capability  Board,  JROC,  and  JS  J-8  (K.  Carlan,  personal 
communication,  April  1 1 , 2012;  D.  Glenister,  personal  communication,  April  16,  2012). 
Some  of  the  passive  stakeholders  include  other  COCOMs,  Services,  defense  agencies, 
combat  support  agencies,  and  inter-agencies.  It  is  important  to  note  that  the  majority  of  the 
Joint  Staff  CGA /  PACOM  IPL  process  stakeholders  interact  with  but  are  not  necessarily 
controlled  by  PACOM. 


Figure  3.  PACOM  IPL  Process  Conceptual/Temporal  Boundaries 

The  JS  CGA/PACOM  IPL  process  is  characterized  by  mixed  boundaries.  The 
boundary  between  the  PROP  process  and  JS-led  CGA  process  is  semi-porous:  Although 
interaction  between  the  PROP  and  the  PACOM  Commander  is  two-way,  there  are  no 
formally  mandated  feedbacks  subsequent  to  PACOM’s  IPL  submission  (D.  Glenister, 
personal  communication,  November  26,  2012).  PACOM  does  not  control  the  JS-led  CGA 
process.  The  outer  boundaries  of  the  JS  CGA/PACOM  IPL  process  present  different  levels 
of  openness.  At  the  entry  point,  the  boundary  is  soft  and  semi-porous,  allowing  interaction 
between  the  PACOM  warfighter  and  the  PROP  (K.  Carlan,  personal  communication,  April 
11,  2012;  December  10,  2012).  At  the  exit  point,  where  the  JROCM  is  completed  and 
distributed  to  COCOMs,  Services,  and  OSD  offices,  the  boundary  is  hard  with  no  two-way 
interaction. 
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Comparative  Assessment 

A  comparison  of  programmatic  boundaries  indicates  that  while  the  FNC  process  has 
clear  programmatic  boundaries  with  specific  funding  lines  and  approved  types  of  actionable 
gaps  (e.g.,  naval  S&T  gaps  with  solutions  at  set  TRLs);  the  JS  CGA/IPL  process  does  not 
have  an  associated  type  of  funding  approved/authorized  and  is  expected  to  address  any 
gap  that  is  deemed  significant.  Similarly,  comparison  of  the  temporal  boundary  reveals  that 
even  though  both  processes  have  a  three-phase  lifecycle  (pre,  during,  and  post;  see  Figures 
2  and  3),  they  differ  in  system  goals.  Unlike  the  FNC  system,  which  is  designed  to  deal  with 
the  full  gap-to-solution  lifecycle,  the  JS  CGA/PACOM  I  PL  process  is  designed  to  deal  with 
only  capability  gaps.  Finally,  stakeholder  boundaries  show  that  the  FNC  process  has  a  more 
comprehensive  approach  to  stakeholder  participation  in  line  with  its  full  lifecycle  process. 
Unlike  the  JS  CGA/PACOM  I  PL  process,  the  FNC  process  includes  not  only  operators  but 
also  resourcers  and  developers.  In  addition,  all  FNC  stakeholders  are  bound  by  both 
organization  (i.e.,  single  naval  command)  and  process,  which  ensures  a  unified 
organizational  vision  and  direct  accountability.  The  JS  CGA/PACOM  I  PL  process,  on  the 
other  hand,  predominately  involves  representatives  from  the  operational  communities  (with 
limited  input  from  the  acquisition  community)  who  are  bound  by  process.  These 
stakeholders — controlled  by  multiple  command  authority — have  competing  visions  and 
distributed  accountability.  The  nature  of  the  boundaries  also  impacts  the  quality  of 
stakeholder  communications.  Even  though  both  processes  have  semi-porous  boundaries, 
FNC  relies  on  a  collaborative  information  exchange  (i.e.,  two-way  communications)  while  the 
JS  CGA/PACOM  IPL  process  involves  predominantly  one-way  information  delivery. 

Key  Insights 

Consequences  resulting  from  differences  in  system  boundaries  of  these  two 
processes  include  the  following: 

•  The  FNC  process  is  designed  to  maximize  gap  resolution  through  dedicated 
programs. 

•  The  JS  CGA/IPL  system  defines  gaps  but  is  not  equipped  to  buy,  develop, 
and  produce  solutions. 

•  Unified  command  structure  allows  the  FNC  system  to  centralize  control  while 
the  JS  CGA/PACOM  IPL  process  has  distributed  and  decentralized  control. 

•  Broad  program  definition  (all  gaps  by  all  COCOMs)  introduces  complexity  into 
the  JS  CGA/PACOM  IPL  process. 

•  Collaborative  communication  enhances  shared  understanding  and 
expectations  amongst  FNC  stakeholders. 

Triplet  2:  Wholes,  Parts,  Relationships 

Navy’s  FNC  Process 

The  FNC  system  is  comprised  of  its  mission,  stakeholders,  processes,  S&T  gaps, 
the  S&T  budget,  S&T  products,  FNC  pillars,  Enabling  Capability  (EC)  proposals,  and  ECs. 
When  these  parts  and  their  relationships  come  together,  they  form  a  coherent  and 
meaningful  whole — the  FNC  system.  For  example,  ECs  are  made  up  of  S&T  products, 
which  respond  to  S&T  gaps.  Similarly,  ECs  are  organized  along  the  nine  FNC  pillars. 

Analysis  of  this  triplet  shows  that  the  FNC  process  constitutes  a  well-organized 
system.  Stakeholders  have  a  clear  understanding  of  the  parts  and  the  overall  system 
mission.  Moreover,  the  relationships  between  stakeholders  are  facilitated  by  codification  of 
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roles  and  responsibilities.  The  parts  are  also  knit  together  with  effective  flow  of  information, 
resources,  and  activities.  As  such,  they  are  configured  to  form  a  well-connected  whole. 

Joint  Staff  CGA/PACOM I  PL  Process 

The  IPL  system  as  a  whole  is  made  up  of  stakeholders,  processes,  PACOM 
operational  shortfalls,  PACOM  IPL  of  capability  gaps,  and  Joint  capability  gaps.  As  an 
example,  PACOM  components  identify  and  prioritize  PACOM  operational  capability  gaps, 
which  are  the  basis  for  the  PACOM  IPL.  PACOM  IPL  capability  gaps  are  then  evaluated  for 
and  may  be  incorporated  into  joint  capability  gaps. 

A  key  observation  from  this  triplet  is  that  PACOM’s  IPL  development  process  is 
dependent  on,  but  does  not  control,  all  parts  participating  in  the  process.  This  is  significant 
for  two  reasons:  (1)  JS,  which  does  not  participate  in  PACOM’s  IPL  development  process, 
controls  the  second  part  of  the  JS  CGA/PACOM  IPL  process;  and  (2)  Joint  capability  gap 
descriptions  are  affected  by  other  COCOM  capability  gaps.  Analysis  of  this  triplet  also  sheds 
light  on  the  flow  of  information  within  PACOM’s  IPL  process.  While  PROP  relationships 
benefit  from  a  collaborative  environment,  PACOM’s  PROP  to  JS  relationship  is 
characterized  largely  through  one-way  interactions.  Additionally,  there  is  limited  reporting  or 
feedback  on  the  status  of  solutions  to  PACOM’s  capability  gaps;  in  addition,  access  to  JS 
tracked  information  requires  that  COCOMs  pull  for  information,  rather  than  receive  it  by  way 
of  JS  push  mechanisms  (D.  Glenister,  personal  communications,  November  26,  2012). 

Comparative  Assessments 

A  comparative  assessment  of  parts  within  the  FNC  and  JS  CGA/PACOM  IPL 
systems  shows  two  discrepancies.  First,  the  FNC  process  can  be  characterized  as  a  single 
system,  whereas  the  JS  CGA/PACOM  IPL  process  merges  two  distinct  systems — that  is, 
PROP  and  CGA  (which  includes  other  COCOM  gap  development  processes) — to  create  a 
new  system.  Second,  the  FNC  process  identifies,  develops,  and  pursues  specific  solutions. 
The  JS  CGA/IPL  process  does  not  involve  specifically  designated  budget  and  other  process 
parts  that  target  solution  (or  product)  development  (K.  Carlan,  personal  communications, 
April  1 1 , 201 2).  Solution  development  is  limited  to  the  FCBs’  preliminary  investigations  on 
potential  solution  sets.9  A  closer  look  at  these  two  processes  reveals  a  number  of 
differences  in  underlying  communication  approaches.  FNC  supports  feedback  relationships 
that  inform  participants  throughout  the  process  and  allow  changes,  as  required.  The  JS 
CGA/PACOM  IPL  process,  on  the  other  hand,  has  limited  feedback,  generally  in  the  form  of 
a  briefing  to  stakeholders  about  final  Joint  gaps,  with  limited  opportunity  for  follow-on 
changes.  In  the  case  of  the  JS  CGA /  PACOM  IPL  process  this  communication  approach  has 
three  implications:  (1)  It  degrades  efforts  to  maintain  continuity  of  effort  and  intent  from 
PACOM  Commander  through  the  final  JS  outcome,  (2)  stakeholders  may  have  limited 
opportunity  to  influence  outcomes  and  may  not  fully  understand  reasoning  behind  final 
decisions,  and  (3)  the  ability  of  the  process  to  self-correct  based  on  process  outcomes  is 
limited. 

Key  Insights 

Some  of  the  important  consequences  of  these  differences  are  listed  here: 

•  The  JS  CGA/PACOM  IPL  process  combines  multiple  systems  and 

stakeholders,  creating  a  potential  for  operational  strife  and  inefficiencies. 
Achieving  process  integration  and  cohesiveness  requires  ongoing,  two-way 
communication,  which,  currently,  is  limited. 


9  The  COCOMs  list  known  On  Going  Efforts  (OGE)  and  recommended  actions/solution,  and  the  FCBs 
request  Service  input  to  review  and  comment  on  COCOM  input  and  provide  their  own  input. 
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•  FNC  solution  investigations  are  more  rigorous  than  those  in  the  JS 
CGA/PACOM  I  PL  process  due  to  follow-on  pursuit  of  product  development. 

•  Current  communication  relationships  between  PACOM,  JS,  and,  ultimately, 
Services  may  lead  to  different  (perhaps  conflicting) 
understandings/expectations  amongst  stakeholders. 

•  In  the  JS  CGA / 1  PL  process,  current  procedures  for  reporting  on  solution 
status  may  curb  shared  understanding  of  how  individual  COCOM  gaps  will  be 
resolved. 

Triplet  3:  Structure,  Function,  Process 
Navy’s  FNC  Process 

The  intended  outcome  of  the  FNC  process  is  to  identify  and  validate  an  annual  list  of 
operational  capability  gaps  and  to  develop  solutions  that  can  be  transitioned  to  a  program  of 
record.  There  are  five  primary  organizational  structures  that  participate  in  the  process:  nine 
IPTs,  which  are  organized  along  nine  FNC  pillars;  the  TOG;  a  Resource  Sponsor;  Program 
Executive  Office  Systems  Command  (PEO/SYSCOM);  and  S&T  (Office  of  Naval  Research, 
2012).  Their  configuration  and  participation  is  determined  according  to  long-standing  FNC 
procedures  and  practices.  Each  organization  performs  a  unique  set  of  roles  and 
responsibilities,  producing  well-defined  information  and  capability  solution  deliverables. 

The  FNC  process  involves  seven  functions  performed  by  the  aforementioned 
organizations  in  the  following  manner,  as  explained  during  FNC  training  (Office  of  Naval 
Research,  2012):  Subsequent  to  the  review  of  operational  problems,  the  nine  IPTs  develop 
and  forward  their  top  three  non-prioritized  S&T  gaps  to  the  TOG.  The  TOG  reviews  and 
approves  the  S&T  gaps  and  forwards  them  to  OPNAV.  OPNAV  officially  promulgates  and 
issues  the  gaps  to  ONR  for  development  and  prioritization  of  ECs.  EC  proposals  are 
developed  through  collaborative  discussions  with  applicable  IPTs.  Furthermore,  Technology 
Transition  Agreements  (TTAs),  which  serve  as  integral  components  of  the  ECs,  are 
developed  by  EC  project  managers.  The  ECs  are  then  forwarded  to  the  IPTs  for  overall 
prioritization.  The  prioritized  ECs  are  reviewed  and  consolidated  by  the  TOG  Working 
Group  and  provided  to  the  TOG  for  approval.  The  TOG  forwards  the  ECs  to  the  S&T 
Sponsor  who  then  submits  the  approved  ECs  as  the  Sponsor  Program  Proposal  (SPP)  to 
the  Chief  of  Naval  Research  for  implementation  through  6.2  and  6.3  S&T  projects. 

Oversight  and  review  is  performed  on  an  ongoing  basis  through  TOG  and  IPT  meetings  as 
well  as  EC  project  execution,  product  development,  and  the  tracking  of  transition  to  POR 
and  to  the  warfighter. 

Joint  Staff  CGA/PACOM  I  PL  Process 

The  primary  intended  outcome  of  the  JS  CGA/PACOM  I  PL  process  is  to  identify, 
prioritize,  and  validate  an  annual  prioritized  list  of  operational  capability  gaps.  It  is 
distributed  amongst  multiple  stakeholders  who  own  different  stages  of  the  process.  The  six 
primary  stakeholders  of  the  process  are  the  PACOM  PROP  components,  PACOM 
Commander,  Joint  Staff  J-8,  FCBs,  JCB,  and  JROC.  They  are  configured  and  linked 
according  to  long-standing  Joint  Staff  and  PACOM  practices.  These  organizations  operate 
in  a  sequential,  and  at  times,  interdependent  manner  to  annually  assess  COCOM  capability 
gaps  along  with  a  preliminary  review  of  solutions.  Each  organization  performs  a  unique  set 
of  roles  and  responsibilities,  which  can  contribute  at  times  to  fragmentation  between  Joint 
Staff  and  PACOM  stakeholders. 

Seven  I  PL  functions  are  performed  by  these  organizations  in  the  following  manner 
(as  confirmed  in  personal  communications;  K.  Carlan,  personal  communication,  April  11, 
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2012):  The  PACOM  PROP  organizations,  which  include  USF-J,  USFK,  USARPAC, 

PACFLT,  MARFORPAC,  PACAF,  SOCPAC,  JIATF-West,  JPAC,  and  ALCOM,  identify, 
organize  and  prioritize  their  organization’s  operational  capability  gaps  and  shortfalls. 
Operational  officers  have  responsibility  to  present  and  convey  their  organization’s 
warfighting  needs  through  collaborative  data  entry  and  dialogue.  Commander’s  guidance, 
Defense  Readiness  Review  System  (DRRS)  deficiencies,  mission  and  associated 
operational  context,  operational  requirements,  and  available  technology  are  all  examples  of 
informational  inputs  to  the  PROP.  PACOM  J-8  facilitates  PROP  and  performs  joint  analysis 
across  COCOM  missions  and  operations.  It  leads  the  formulation,  development,  and 
coordination  of  the  PACOM  IPL.  Utilizing  PROP  data  along  with  other  current  efforts  (e.g., 
Issue  Nominations,  DDRS  deficiencies,  rebalance  initiatives),  PACOM  J-8  prepares  and 
submits  the  recommended  IPL  to  PACOM  key  0-6  level  staff  officers,  J-Code  Directors,  and 
the  PACOM  Deputy  Commander  for  review,  and  incorporates  feedback  in  preparation  for 
submission  to  the  PACOM  Commander.  The  IPL  is  submitted  to  the  PACOM  Commander 
for  comment  and  approval.  The  approved  IPL  is  submitted  to  Joint  Staff  J-8  through  the 
Comprehensive  Joint  Assessment  (CJA).  The  JS  J-8  is  responsible  for  conducting  the 
annual  capability  gap  assessment,  which  is  guided  by  national  military  strategy,  and 
informed  by  IPLs  and  other  warfighting  operational  needs.  The  IPL  capability  gaps  are 
further  defined,  taking  into  account  similarities  and  differences  across  the  COCOMs,  thereby 
creating  a  greater  overall  number  of  gaps  beyond  the  top  90  submitted  by  the  nine 
COCOMs.  J-8  selectively  distributes  the  capability  gaps  to  the  nine  FCBs  for  review.  The 
FCBs  are  responsible  for  assessing  and  confirming  the  relevance  of  the  capability  gaps  and 
identifying  preliminary  capability  solutions,  if  known  and  available.  The  FCBs  facilitate 
collaborative  information  exchange  across  the  Services,  OSD,  non-Service  organizations, 
and  COCOMs.  FOB  results  are  conveyed  to  the  J-8  through  purple  slides  and  quad  charts. 
The  JS  J-8  develops  the  recommended  list  of  prioritized  capability  gaps  for  review  by  the 
JOB.  The  JOB  reviews  and  establishes  the  recommended  prioritized  list  of  capability  gaps 
with  programmatic  recommendations  and  associated  military  and  strategic  risk,  and  submits 
them  to  the  JROC.  The  JROC  validates  the  capability  gaps  for  the  DoD,  and  publishes  a 
JROCM  CGA,  which  is  then  distributed  to  the  USD(AT&L)  and  Services  for  capability 
solutions  acquisition  and  transition  to  the  COCOMs. 

Comparative  Assessment 

Both  the  FNC  and  JS  CGA/PACOM  IPL  processes  are  driven  by  a  number  of 
informational  inputs  such  as  warfighting  requirements  and  operational  plans.  The  processes 
vary  in  their  final  outcomes,  whereby  technology  transition  to  programs  of  record  is  the 
tracked  and  measured  outcome  of  the  FNC  process  while  validated  DoD  capability  gaps  are 
the  final  outcome  of  the  JS  CGA/PACOM  IPL  process. 

Designed  for  a  single  Service,  the  FNC  process  is  a  cohesive  and  well-integrated 
process.  It  is  employed  exclusively  through  the  naval  chain  of  command  and  the  associated 
procedure.  By  contrast,  the  JS  CGA/PACOM  IPL  process  is  divided  along  functions 
executed  by  the  Joint  Staff,  and  multiple  COCOM  and  Service  stakeholders,  who  are  guided 
by  varying  interests,  needs,  and  resources.  As  such,  process  accountability,  traceability,  and 
measures  of  success  are  inevitably  different  within  each  process.  FNC  activities  are 
comprehensive;  FNC  represents  a  “closed-loop”  approach  to  the  process,  enabling 
measurable  outcomes.  In  contrast,  it  is  difficult  for  JS  CGA/PACOM  IPL  stakeholders  to 
quantify  how  many,  and  to  what  degree,  gaps  are  resolved  as  a  result  of  the  JS  CGA/IPL 
process.  Furthermore,  unlike  in  the  FNC  process,  in  the  JS  CGA/PACOM  IPL  process, 
PACOM  capability  gaps  can  lose  specificity,  be  marginalized,  or  may  be  disregarded. 
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Key  Insights 

Some  important  consequences  of  these  differences  are  listed  here: 

•  The  way  COCOM  gaps  evolve  during  the  JS  CGA/PACOM  I  PL  process 
results  in  difficulty  tracking  program  success  as  capability  gaps  and  capability 
results  are  difficult  to  connect. 

•  As  a  solution-  and  product-driven  process,  FNC  establishes  a  robust  and 
transparent  connection  between  gaps  and  solutions,  which  facilitates  tracking 
program  success. 

•  The  JS  CGA/IPL  process  is  designed  to  focus  on  gap  development  and 
validation;  as  such,  it  only  tangentially  addresses  gap-to-solution  outcomes. 

•  The  hard  boundary  between  capability  gap  assessment  and  solutions  may 
limit  JS  CGA/PACOM  I  PL  stakeholder  access  to  information  on  program 
status  and  results. 

Triplet  4:  Inputs,  Outputs,  Transformations 
Navy’s  FNC  Process 

The  FNC  process  is  designed  to  respond  strictly  to  naval  gaps  as  guided  by  naval 
requirements,  sources,  and  mission.  Considering  all  the  activities  that  are  taking  place  within 
the  FNC  system,  two  different  transformations  are  noted:  “Requirements  to  Products”  and 
“Disparate  Perspectives  to  Collective  Ownership”  (Figure  4).  “Requirements  to  Products”  is 
a  systems  engineering  transformation  and,  as  such,  is  readily  visible  with  a  tangible  output. 
FNC  employs  a  full  systems  engineering  lifecycle,  creating  a  seamless,  transparent,  and 
traceable  transformation  from  gap  to  capability.  Transition  of  enabling  capabilities  from  FNC 
is  tracked  from  early  development  activities  through  delivery  to  the  naval  warfighter.  In  this 
transformation,  requirements  are  turned  into  products  to  be  transitioned  to  Acquisition  POR. 
These  products  are  eventually  deployed  to  the  warfighter  to  resolve  a  capability  gap. 

Clearly,  this  transformation  is  critical  for  the  mission  of  the  FNC  program  and  the  broader 
transformation  of  requirements  to  naval  capabilities. 

The  second  transformation  of  “Disparate  Perspectives  to  Collective  Ownership” 
(Figure  4)  is  perhaps  less  noticeable  but  very  critical  in  making  the  FNC  process  a 
successful  one.  It  is  concerned  with  the  evolution  of  attitudes  and  a  culture  of  commitment 
amongst  disparate  stakeholders.  The  FNC  process  has  institutionalized  mechanisms  to 
create  a  collaborative  environment,  establish  clear  roles  and  responsibilities,  and  reinforce  a 
culture  of  collaboration  and  accountability.  Collaborative  participation  platforms,  TTAs,  and 
Transition  Commitment  Levels  (TCLs)  all  commit  stakeholders  to  clearly  and  explicitly  stated 
role  assignments,  transforming  functional  area  biases  of  requirements/acquisition 
communities  into  an  integrated  viewpoint  and  a  shared  mission  amongst  distinct 
stakeholders.  This  collective  understanding  of  responsibility,  accountability,  and  ownership, 
in  turn,  enables  the  “Requirements  to  Products”  transformation  referenced  previously. 
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Joint  Staff  CGA/PACOM I  PL  Process 

The  JS  CGA/PACOM  IPL  process  is  designed  to  consider  all  COCOMs’  needs/gaps, 
each  serving  as  an  input  to  the  JS  CGA  process.  Therefore,  diverse  COCOM  capability 
gaps  (determined  by  their  varying  missions)  are  considered  and  prioritized. 

There  are  three  sequential  transformations  for  gap  characterization  identified  within 
the  JS  CGA/PACOM  IPL  process.  The  first  transformation,  “PACOM  Requirements  to 
PACOM  Gaps,”  takes  place  within  PROP.  Based  on  PACOM’s  mission,  existing  capabilities, 
available  technology,  and  lessons  learned,  PACOM  components  consider  PACOM 
requirements  and  provide  the  PACOM  Deputy  Commander  with  capability  shortfalls/gaps. 
During  the  next  transformation,  “PACOM  capability  gaps  to  PACOM  IPL,”  the  PACOM 
Commander  evaluates  PACOM  gaps  and  issues  the  final  PACOM  IPL.  Through  the  third 
transformation,  “PACOM  IPL  to  JROCM  CGA,”  the  PACOM  IPL  is  merged  with  other  inputs 
(e.g.,  other  COCOM  IPLs,  national  military  strategy,  etc.)  to  produce  joint  capability  gaps 
that  are  identified  and  prioritized  in  the  JROCM  CGA. 

The  application  of  this  triplet  to  the  JS  CGA/PACOM  IPL  process  brings  to  the  fore 
the  fact  that  out  of  the  three  transformations,  PACOM  controls  only  the  first  two,  which 
results  in  the  PACOM  IPL.  The  third  transformation  is  controlled  by  JS.  Since  this 
transformation  is  designed  to  integrate  and  prioritize  DoD-wide  capability  gaps,  it  takes  into 
account  all  COCOMs’  gaps  when  producing  final  capability  gap  descriptions.  As  a  result, 
PACOM’s  IPL  is  influenced  by  other  COCOMs’  capability  gaps  and  their  priorities. 

Comparative  Assessments 

As  opposed  to  the  FNC  system,  which  is  more  narrowly  focused  on  naval-only  gaps, 
the  JS  CGA/PACOM  IPL  process  is  designed  to  address  DoD-wide  capability  needs  and 
consider  all  COCOM  capability  gaps.  As  such,  a  diverse  range  of  missions,  actors, 
requirements,  and  procedures  feeds  into  the  JS  process. 

A  comparison  of  the  FNC  and  JS  CGA/PACOM  IPL  process  transformations  shows 
that  FNC  measures  of  success  include  not  only  gap-to-solution  considerations  but  also 
product  transition  to  warfighter.  The  JS  CGA/PACOM  IPL  process,  on  the  other  hand,  does 
not  include  a  gap-to-solution  transformation.  Additionally,  the  JS  CGA/PACOM  IPL  system 
does  not  have  a  formal  mandated  tracking  mechanism  linking  JROCM  outputs  to  solution 
outcomes.  Another  difference  between  the  two  processes  is  that  FNC  transforms  functional 
area  biases  of  the  requirements/acquisition  community  into  a  shared  understanding  and  set 
of  collaborative  relationships  amongst  stakeholders.  While  the  JS  CGA/PACOM  IPL  process 
does  leverage  working  groups  and  IPTs  as  consensus  building  platforms,  it  does  not  have 
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formal  and  binding  agreements  for  reconciling  varying  organizational  cultures  and  norms 
even  though  it  serves  a  more  diverse  set  of  clients  and  needs. 

Key  Insights 

These  differences  have  several  implications  for  the  two  processes’  performances: 

•  PACOM  capability  gap  prioritization  may  be  marginalized  due  to  other 
COCOM  higher  priority  capability  gaps.  Similarly,  PACOM  capability  gaps 
may  lose  theater  operational  specificity  when/if  they  are  merged  with  other 
related  COCOM  capability  gaps. 

•  Due  to  the  impact  of  multiple  COCOM  gaps  on  joint  capability  gaps,  it  is 
difficult  to  effectively  link  individual  IPL  inputs  to  joint  CGA  outputs. 

•  There  is  limited  ability  to  effectively  track  and  measure  the  overall  transition 
success  in  JS-led  IPL  transformation. 

•  Cultural  transformation  in  the  FNC  system  enables  agreement,  shared 
ownership,  and  direct  accountability. 

•  FNC  outcomes  resonate  with  stakeholders  as  they  produce  tangible  products 
as  an  output. 

•  By  nature  of  the  JS  CGA/PACOM  IPL  process,  stakeholder  definitions  of 
successful  transformation  may  vary — system  transformation  may  be 
successful,  but  it  does  not  mean  all  stakeholders  win. 

Triplet  5:  Command,  Control,  Communications 

Navy’s  FNC  Process 

Command  of  the  FNC  process  is  implemented  through  ownership  and  decision¬ 
making  organizations.  The  S&T  Corporate  Board  owns  responsibility  for  the  FNC  program 
and  establishes  policy  as  well  as  the  participating  organizations’  roles  and  responsibilities. 
The  nine  IPTs,  TOG,  and  Chief  of  Naval  Research  (CNR)  have  decision-making  authorities 
for  the  execution  of  the  process,  production  of  the  S&T  gaps,  and  development  and 
transitioning  of  the  ECs  and  solutions  to  the  POR  for  operational  use.  Control  of  the 
process,  organizational  behaviors,  and  rules  of  engagement  are  built-in  mechanisms  at  both 
the  macro  and  micro  levels.  DoD  5000.02  (OUSD[AT&Lj,  2008),  naval  requirements,  and 
the  nine  FNC  pillars  provide  overarching  macro  control  guidance.  The  TOG  Charter,  FNC 
business  rules,  TTAs  (including  TRLs  and  TCLs),  project  plans,  staff  training,  and  transition 
metrics  are  micro-level  controls  to  manage  and  track  performance,  and  support  decision¬ 
making  throughout  the  process  lifecycle. 

The  FNC  process  promotes  and  facilitates  formal  and  informal  collaborative 
information  sharing.  Closed-loop  communications  facilitate  accountability  and  provide  clear 
measurable  data  on  outcomes.  Information  flows  through  organized  coordination  and 
feedback  mechanisms  with  mandatory  participation  requirements  for  all  stakeholders. 
Scheduled  IPT,  TOG,  and  TOG  working  group  meetings  and  informal  information 
exchanges  enable  stakeholders  and  decision-makers  to  assess  and  track  key  S&T  gaps 
and  ECs,  and  transition  capability  solutions  to  the  POR  for  operational  use.  Dual-hatted  IPT 
co-chairs  also  facilitate  information  sharing  and  general  situational  awareness  through  both 
formal  and  informal  channels  as  they  move  between  FNC  and  non-FNC  worlds. 
Documentation  of  the  gaps,  ECs,  project  plans,  TTAs,  and  transition  reports  are  established, 
maintained,  and  shared  across  the  FNC  organizations. 
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Joint  Staff  CGA/PACOM I  PL  Process 

The  JS  CGA/PACOM  I  PL  process  is  not  owned  or  governed  by  a  single  organization 
and  its  stakeholders.  The  JS  owns  responsibility  for  the  DoD  CGA  process,  including  the 
joint  capability  gap  assessment  and  the  JROC  validation.  PACOM  is  responsible  for 
developing,  approving,  and  submitting  the  PACOM  IPL.  The  JROC,  JCB,  JS  J-8,  and 
PACOM  Commander,  have  critical  decision-making  power.  As  part  of  their  roles  and 
responsibilities,  they  make  decisions  to  execute  the  process,  and  develop  and  validate 
capability  gaps.  FCBs,  FCB  working  groups,  and  PROP  organizations  provide  prioritized 
recommendations  to  the  process.  There  are  limited  control  mechanisms  within  the  JS 
CGA/PACOM  IPL  process  to  facilitate  whole-process  integration.  Formal  control  is  limited 
due  to  the  absence  of  such  aspects  as  standard  protocols  or  measures  of  collective 
ownership  and  accountability.  The  DoD  and  PACOM  level  controlling  guidance  includes 
Defense  Planning  and  Programming  Guidance  (DPPG);  CJCSI  8501. 01 A  (CJCS,  2004); 
3110. 01G  (released  in  2008);  3170. 01H  (CJCS,  2012);  JS  guidance  on  how  to  compose  an 
IPL;  Planning,  Programming,  Budgeting  and  Execution  System  (PPBES);  Pacific  Theater 
Strategy  (first  edition);  and  PACOM  PROP  Instruction  (Commander,  U.S.  Pacific  Command 
[USPACOM],  2011). 

Communications  take  place  predominantly  in  the  form  of  one-way  information 
delivery.  They  are  performed  through  written  as  well  as  verbal  information  exchange 
mediums.  The  CGA,  IPL,  purple  slides,  quad  charts,  and  JROCM  Capability  Gap 
Assessment  are  classified  documents  that  communicate  assessed,  prioritized,  and  validated 
capability  gaps.  The  JROC,  JCB,  FCBs  and  working  groups,  J-8  Worldwide  Conference, 
and  PACOM  PROP  serve  as  platforms  that  enable  dialogue  and  discussions  across 
operational  and  programmatic  representatives. 

Comparative  Assessment 

FNC  maintains  command  of  the  full  lifecycle  of  the  process  and  the  associated 
decision-making  since  the  governance  framework  that  is  in  place  is  holistic,  integrated  and 
owned  by  a  single  organization.  The  JS  CGA/PACOM  IPL  process  is,  on  the  other  hand, 
governed  through  multiple  separate  and  independent  command  structures.  PACOM  owns 
its  PACOM  IPL  process  as  evidenced  by  the  Commander’s  approval  and  submission  of  the 
IPL.  Thereafter,  the  JS  governs  the  process  for  the  DoD  and  all  COCOMs.  FNC  is 
controlled  by  macro-level  USN  guidance  and  policies  and  a  micro  view  of  instructions,  as 
well  as  roles  and  responsibilities.  The  JS  CGA/PACOM  IPL  process  relies  mainly  on  macro 
command  level  guidance  driven  by  DoD-wide  needs  and  demands. 

Both  processes  provide  opportunities  for  collaboration  and  coordination.  The  JS 
capability  gap  assessment  is,  nonetheless,  not  dependent  on  COCOM  participation.  As 
such,  COCOM  capability  gaps  may  be  marginalized.  Further,  COCOM  participation  is 
informed  by  real  or  perceived  limited  return  on  time  investment.  FNC  stakeholders  have  a 
clear  understanding  of  the  FNC  lifecycle  process  and  are  provided  specific  updates. 
COCOMs  have  diverging  perceptions  of  the  process  and  may  be  provided  more  ambiguous 
information  regarding  how  joint  gaps  evolve  during  the  cycle  and  are  subsequently  fulfilled. 
FNC  offers  enhanced  accountability  through  its  complete  feedback  loop  that  closes  the 
lifecycle  of  a  gap  by  reporting  outcomes  to  stakeholders.  JS  CGA/PACOM  IPL 
communications  are  hindered  by  logistics,  practical  process  difficulties,  and,  perhaps  most 
importantly,  the  potential  discrepancy  between  the  gap  assessments  by  COCOMs  and  the 
DoD. 
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Key  Insights 

These  differences  between  the  two  processes  generate  the  following  implications: 

•  Centralized  and  integrated  governance  and  participatory  processes  that 
characterize  the  FNC  system  provide  continuity  of  process  by  allowing 
stakeholder  input  and  agreement  throughout  the  lifecycle  of  the  process  and 
by  developing  shared  understanding,  joint  ownership,  and  accountability. 

•  The  distributed  governance  and  one-way  communications  that  characterize 
the  JS  CGA/PACOM  IPL  process  disrupt  the  continuity  of  the  process  and 
provide  limited  opportunities  for  management  of  stakeholder  perceptions  and 
resulting  expectations. 

•  As  a  multi-stakeholder  process,  the  JS  CGA/PACOM  IPL  process  may  need 
formal  integration  mechanisms  to  bring  different  parts  of  the  process  together 
for  shared  ownership  and  enhanced  accountability. 

•  The  absence  of  formal  tracking  of  gap  modification  and  related  outcomes 
limits  JS  CGA/PACOM  IPL  process  feedback  and  creates  the  potential  for 
missed  opportunities  in  adjusting/correcting  performance. 

Triplet  6:  Openness,  Hierarchy,  Emergence 

Navy  FNC  Process 

The  FNC  system  presents  two-way  openness  throughout  its  process  lifecycle: 
exterior  to  interior  and  interior  to  exterior.  Exterior  to  interior  openness  is  evidenced  by  the 
system’s  ability  to  accept  evolving  naval  requirements/gaps,  advances/availabilities  of 
technology,  new  S&T  developers,  and  transition  venues/methods.  The  FNC  process  is  also 
responsive  to  data  on  transition  to  the  warfighter.  Interior  to  exterior  openness  is  presented 
by  the  dual-hatted  FNC  players’  ability  to  cross  into  non-FNC  domains. 

There  is  some  evidence  of  re-configurability  of  system  hierarchy.  For  example,  “the 
FNC  pillars  were  reduced  from  1 1  to  5  in  2004”  (Goldstein,  2006).  Similarly,  in  2005  the  FNC 
system  “was  restructured  to  align  with  the  pillars  of  the  Chief  of  Naval  Operations’  and  the 
Commandant  of  the  Marine  Corps’  vision  for  the  future — Naval  Power  21 — and  to  focus  on 
providing  Enabling  Capabilities  to  close  warfighting  gaps”  (Office  of  Naval  Research,  n.d.- 
b).10 

The  FNC  system  accommodates  changing  circumstances  and  new  process  parts;  it 
allows  itself  to  be  open  to  emergent  system  properties.  For  example,  the  transition  data 
influence  subsequent  FNC  processes  and  performance,  which  facilitate  resource-efficient 
management  of  FNC  as  it  permits  revision  of  the  process  and  adjustments  in  performance 
based  on  transition  statistics. 

Joint  Staff  CGA/  PACOM  IPL  Process 

The  JS  CGA/PACOM  IPL  process  is  quasi-open  and  presents  mostly  one-way 
openness.  Exterior  to  interior  openness  of  the  JS  CGA/PACOM  IPL  process  can  be  seen  in 
its  acceptance  of  evolving  requirements  and  gaps.  However,  the  process  is  closed  to 
stakeholder  feedback  subsequent  to  the  issuance  of  the  JROCM.  Interior  to  exterior 
openness  is  evidenced  by  the  interior  stakeholder  communications  with  the  exterior 
stakeholders. 


10  This  quote  can  be  found  on  the  Office  of  Naval  Research  website  as  cached  by  December  9,  2012. 
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The  JS  CGA/PACOM  I  PL  process  has  limited  options  for  change  as  the 
configuration  of  system  elements  appears  stable.  As  a  result  of  limited  re-configurability,  the 
system  is  less  versatile  and  is  at  risk  of  critique  and  perception  of  ineffectiveness  by 
stakeholders. 

Comparative  Assessments 

Both  FNC  and  JS  CGA/PACOM  IPL  processes  present  openness,  but  the  JS 
CGA/PACOM  IPL  process’  openness  terminates  subsequent  to  the  JROCM  conveyance. 
Another  critical  distinction  between  the  two  processes  is  concerned  with  tracking  of 
transition  data.  The  FNC  process  is  responsive  to  data  on  transition  to  the  acquisition  POR 
and  eventual  deployment  of  the  S&T  product  or  capability  to  the  fleet.  The  JS  CGA/PACOM 
IPL  process  tracks  CGA  gaps  (i.e.,  COCOM  gaps  that  are  accepted  as  joint  gaps),  but  does 
not  formally  track  other  aspects  of  the  process  (i.e.,  COCOM  gap  modification  and  matching 
of  outcome  to  original  COCOM  gap). 

Key  Insights 

Some  of  the  critical  consequences  that  emerge  out  of  the  previous  discussion 
include  the  following: 

•  Given  there  is  not  a  mandated  feedback  system  nor  a  comprehensive 
transition  tracking  system  in  the  JS  CGA/PACOM  IPL  process,  stakeholders 
may  pursue  what  they  see  as  unfilled  capability  gaps  through  other  means 
such  as  JCTDs  and  JUONS,  leading  to  a  potential  duplication  of  efforts  and 
inefficient  use  of  resources. 

•  The  FNC  process  leverages  transition  data  to  modify  subsequent  FNC 
process  and  performance.  This  may  result  in  a  more  resource-efficient 
management  of  the  process. 

Triplet  7:  Variety,  Harmony,  Parsimony11 

Navy’s  FNC  Process 

The  FNC  process  introduces  an  instance  of  harmony  in  the  way  it  deals  with  the 
tracking  of  S&T  products  and  their  delivery  to  the  naval  warfighter.  To  achieve  harmony,  the 
FNC  process  tracks  two  important  transitions.  First,  the  process  tracks  the  transition  of  S&T 
products  to  the  intended  acquisition  POR  (Office  of  Naval  Research,  2012).  As  noted  in  the 
boundaries  discussion,  this  transition  is  an  immediate  output  of  the  FNC  process,  and  is 
therefore  tracked  as  a  critical  success  metric.  Even  so,  tracking  FNC  product  transition  to 
the  POR  does  not  necessarily  provide  a  complete  and  accurate  assessment  of  product 
delivery  to  the  warfighter.  For  example,  the  POR  itself  may  never  transition  to  the  warfighter 
due  to  a  number  of  factors,  including  program  cancellation.  Similarly,  transitioning  products 
to  a  POR  is  not  an  end  in  itself;  rather  it  serves  a  broader  purpose.  In  this  regard,  tracking 
only  transition  to  the  POR  runs  the  risk  of  creating  a  false  sense  of  S&T  integration  success 
and  is,  therefore,  too  parsimonious  of  a  view.  To  counter  this,  FNC  leverages  variety  in 
transition  statistics.  In  addition  to  tracking  transition  to  the  POR,  the  FNC  program  tracks 
POR  transition  to  the  warfighter  (Office  of  Naval  Research,  2012).  Just  as  tracking  the 
transition  of  products  to  the  POR  risks  creating  a  false  sense  of  success,  tracking  only 
transition  of  PORs  to  the  warfighter  runs  the  risk  of  creating  an  inaccurate  assessment  of 


11  The  triplet  of  variety,  harmony  and  parsimony  is  probably  the  most  abstract  Conceptagon  triplet  of 
all.  Rather  than  identifying  every  instance  of  variety  and  every  instance  of  parsimony,  the  study  team 
sought  to  identify  instances  of  harmony,  as  these  were  believed  to  indicate  practices  that  enhanced 
system  performance. 
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S&T  product  failure.  This  is  primarily  because  S&T  products  are  but  one  of  many  factors  in 
determining  a  POR’s  successful  transition.  It  is  the  balance  of  viewing  success  of  direct 
outputs  of  the  FNC  process  (S&T  product  transition  to  POR)  through  the  lens  of  broader 
naval  capability  delivery  (how  and  if  those  products  are  ultimately  used  by  the  warfighter) 
that  brings  harmony  to  the  FNC  program  assessments. 

Joint  Staff  CGA/PACOM I  PL  Process 

By  nature  of  its  service  to  multiple  stakeholders  across  all  COCOMs,  the  JS 
CGA/PACOM  IPL  process  is  subject  to  great  variety.  An  instance  of  variety  is  found  in  the 
number  of  number-one  priority  gaps  submitted  to  the  Joint  Staff.  Herein  COCOMs  submit 
their  top  gaps  (through  each  COCOM’s  IPL)  to  the  Joint  Staff  (later  forwarded  to  the  JROC) 
for  inclusion  in  the  final  CGA.  Individual  COCOM  gaps  are  ranked  in  priority  order  preceding 
submission  to  the  JS  J-8.  This  process  is  conducive  to  producing  much  variety  in  the 
number  of  incoming  number-one  priority  gaps,  as  all  nine  COCOMs  submit  a  gap 
designated  as  number-one.  The  JS  J-8  receives  the  ranked  gaps  from  each  of  the  COCOMs 
and  then  works  to  combine  them  into  a  new,  consolidated  list  of  prioritized  gaps.  This 
consolidation  is  done  so  that  the  DoD  can  work  from  a  single  list  of  prioritized  defense  gaps. 
The  process  is  also  used  to  reduce  duplication  of  gaps  and  to  combine  similar  gaps.  The  JS 
J-8  list  is  then  forwarded  through  the  FCBs  and  JCB,  wherein  each  group  reviews  and 
reorganizes  (as  required)  the  list  of  prioritized  gaps  prior  to  final  review  and  approval  by  the 
JROC.  The  passing  of  the  list  through  many  review  boards  also  introduces  variety  in  the 
process. 

Although  the  processes  of  combining,  accepting,  rejecting,  realigning,  and 
reorganizing  COCOM  gaps  into  a  single  finalized  list  of  joint  capability  gaps  does  introduce 
parsimony  (in  the  creation  of  a  consolidated  list),  it  does  not  necessarily  bring  about 
harmony.  By  requesting  a  ranked  list  of  gaps  from  individual  COCOMs,  the  process 
inadvertently  sets  an  expectation  that  the  number-one  gaps  should  all  receive  top  priority  in 
the  final  list.  However  this  may  not  be  the  case  as  one  COCOM’s  top  ten  needs  may  make 
the  final  list  while  another  COCOM  may  not  see  any  of  its  submitted  gaps  reflected  in  the 
final  CGA.  Furthermore,  the  process  of  homogenizing  many  seemingly  similar  gaps  into  one 
broad  gap  statement  may  create  a  gap  so  broad  that  eventual  solutions  fail  to  meet  the 
original  need  of  the  submitting  COCOM(s).  Additionally,  variety  in  the  number  of  reviews  as 
well  as  groups  responsible  for  approving  the  CGA  fragments  the  ownership  process. 

Comparative  Assessment 

Both  FNC  and  the  Joint  Staff  processes  accept  gap  statements  from  various  groups 
(IPTs  and  COCOMs,  respectively);  however,  by  consolidating  and  reprioritizing  COCOM- 
submitted  number-one  needs,  the  Joint  Staff  process  may  result  in  stakeholder  frustration. 
The  FNC  process  benefits  from  a  more  direct  coordination  process  as  opposed  to  the  Joint 
Staff  process  where  gaps  must  travel  through  many  working  groups  and  boards.  With  each 
change  of  hands  in  the  Joint  Staff  process,  gaps  are  further  homogenized,  refined,  and 
altered.  This  extended  process  of  shaping  capability  gaps  can  result  in  gaps  that,  in  some 
cases,  only  marginally  reflect  the  originally  submitted  COCOM  need. 

Key  Insights 

There  are  two  major  consequences  and  their  associated  ripple  effects  resulting  from 
these  instances  of  variety,  harmony,  and  parsimony: 

•  The  FNC  process  achieves  harmony  at  multiple  stages  of  the  process 

through  its  organizational  structure  and  continuous  planning  for  final  transition 
to  the  warfighter. 
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•  Fragmentation  of  reviewers  (through  the  JS  process)  and  homogenization  of 
COCOM  IPLs  into  joint  capability  gaps  may  cause  discord  leading  to  general 
solutions  that  do  not  effectively  satisfy  specific  COCOM  needs;  difficulty  in 
identifying  and  tracking  original  COCOM  gaps;  and  COCOMs’  resubmission 
of  the  same  gaps  in  subsequent  years. 

Conclusions,  Recommendations,  and  Strategies  for  Improvements 

Admittedly,  endstates  of  the  two  systems  (the  naval  FNC  process  and  the  JS 
CGA/PACOM  I  PL  process)  are  markedly  different;  FNC  is  a  gap-to-solution  process 
(including  both  gap  and  solution  development  phases)  whereas  the  JS  CGA /  PACOM  IPL 
process  is  solely  a  gap  development  process.  In  addition  to  varying  endstates,  the  systems 
differ  in  the  breadth  of  customers  they  serve.  FNC,  a  naval  program  serving  naval 
warfighters,  is  united  by  purpose,  mission,  Service,  and  ownership — in  other  words,  it  is  an 
end-to-end  by  Navy/for  Navy  process.  On  the  other  hand,  the  JS  process  serves  multiple 
customers  (e.g.,  COCOMs,  Services).  As  such,  while  individual  COCOM  IPL  development 
processes  may  be  united  by  mission  (though  not  Service),  the  final  disposition  of  CGA  will 
be  subject  to  a  variety  of  needs  across  different  missions  and  purposes. 

Bearing  these  differences  in  mind — particularly  as  related  to  varying  endstates — we 
focused  our  comparison  on  steps  in  both  processes  that  lead  to  a  common  milestone  of 
final,  prioritized  gaps.  In  the  course  of  the  comparison,  it  became  evident  that  FNC’s 
integration  of  the  solution  phase  better  positions  the  FNC  process  to  leverage  inherent 
feedbacks  between  the  two  phases,  thereby  enhancing  its  responsiveness  to  naval  needs. 
Although  solution  development  activities  will  remain  with  the  Services,  our  analysis  indicates 
the  JS  CGA/PACOM  IPL  process  may  benefit  from  strategies  employed  by  the  FNC 
program  such  as  communication  and  comprehensive  gap-to-solution  tracking  strategies. 

The  team  further  believes  the  following  recommendations  may  serve  to  mitigate  some  of  the 
frustrations  experienced  by  stakeholders  in  the  JS  CGA/PACOM  IPL  process. 

•  The  JS  CGA/PACOM  IPL  process  could  expand  its  boundary  to  include 
solution  providers.  Existing  structures  (e.g.,  FCBs,  supporting  working 
groups)  can  be  used  to  facilitate  formal  participation  by  the  acquisition 
community. 

•  The  JS  could  improve  transparency  relative  to  gap  status.  Formal, 
accountable,  ongoing  and  two-way  communication  will  facilitate 
understanding,  expectation  management,  and  feedback. 

•  Gap  and  solution  progress  statistics,  collected  in  coordination  with  the 
acquisition  and  warfighter  communities  (currently  partially  tracked  by  the  JS 
and  accessible  through  KMDS),  could  be  documented  and  published 
periodically  (e.g.,  annually  or  bi-annually)  and  could  be  actively  disseminated 
to  COCOMs  and  briefed  at  FCBs  and  JCBs. 

•  The  JS  and  PACOM  should  leverage  tracking  metrics  (as  recommended 
above)  to  inform  process  improvement  efforts.  The  JS  process  should 
establish  procedures  and  better  define  roles  and  responsibilities  to  encourage 
metric  and  process  accountability. 

•  The  JS  can  improve  stakeholder  satisfaction  and  commitment  by 
promulgating  process  steps  and  clarifying  guidance  documents.  The  JS 
should  update  related  Instructions  and  should  consider  developing  user 
friendly  products  and  reports  that  explain  the  process. 
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Abstract 

Acquiring  defense  systems  in  the  face  of  urgent  needs,  budget  challenges,  and  scarce 
resources  drives  the  need  for  designing  affordable  systems  by  anticipating  uncertain  futures 
and  related  mission  volatility.  Despite  recent  strategies  to  mandate  designing  for  affordability 
as  a  reguirement  in  acquisition  management,  current  processes  for  performing  early  lifecycle 
affordability  tradeoffs  remain  underdeveloped.  Methods  for  exploring  design  tradespaces 
have  matured  but  have  lacked  specific  focus  on  evaluating  affordability.  Affordability  tradeoffs 
have  also  largely  been  limited  to  static  tradeoffs  of  systems  in  current  operating 
environments,  or  in  single  point  futures.  Given  that  systems  exist  in  a  dynamic  and  uncertain 
world,  designing  for  affordability  necessitates  a  method  capable  of  evaluating  systems  across 
many  possible  alternative  futures.  A  new  method  that  leverages  both  Multi-Attribute 
Tradespace  Exploration  and  Epoch-Era  Analysis  is  proposed  to  augment  current  practices  in 
designing  for  affordability.  This  method  can  be  applied  to  the  evaluation  of  system  concepts 
across  multiple  epochs  (periods  of  fixed  context  and  needs)  and  multiple  eras  (ordered 
sequences  of  epochs)  and  uses  the  Multi-Attribute  Expense  metric  to  provide  a  measure  for 
aggregate  costs  and  schedule  considerations.  This  paper  presents  interim  research 
outcomes  of  an  ongoing  research  project,  demonstrating  the  viability  of  the  approach  with  a 
case  study. 
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Motivation 

With  recent  advances  in  technology  and  growing  demand  for  enhanced  warfighter 
capabilities,  defense  acquisition  programs  have  become  increasingly  complex  and  costly  to 
manage  in  the  long  term.  Since  the  Department  of  Defense  (DoD)  is  responsible  for  the 
research,  development,  production,  and  delivery  of  weapon  systems  on  time  and  at  a 
reasonable  cost,  there  is  an  emergent  need  for  the  DoD  to  manage  the  cost,  schedule,  and 
performance  of  a  major  weapon  system  or  program  that  typically  runs  into  billions  of  dollars 
within  its  lifecycle. 

With  the  prospect  of  slowly  growing  or  flat  defense  budgets  in  coming  fiscal  years 
(GAO,  2011)  and  recent  budget  cuts,  the  DoD  is  seeking  ways  to  yield  better  returns  on  its 
weapon  system  investments  and  find  methods  of  delivering  defense  capabilities  for  less 
than  it  has  in  the  past.  It  is  well  understood  that  mitigating  uncertainty  in  estimating  cost  and 
schedule  parameters  that  plague  the  early  phases  of  program  formulation  would  help  to 
identify  the  true  costs  of  a  weapon  system  or  program  from  the  beginning  and  reduce 
overruns.  This  is  also  in  consonance  with  the  advent  of  capability-based  planning,  which 
aims  to  counter  external  threats  with  the  best  warfighter  capabilities  deliverable  under 
constrained  economic  conditions  and  uncertainty  (Patterson,  2012). 

Efforts  to  improve  cost  and  schedule  estimation  are  ongoing,  but  there  has  been 
relatively  little  progress  in  addressing  uncertainties  related  to  costs  stemming  from 
alternative  futures  that  the  system  may  face.  The  research  described  in  this  paper  is 
motivated  by  this  specific  aspect  in  the  increasingly  urgent  need  of  designing  for 
affordability. 

Background 

Buying  strategies  are  continuously  evolving  to  place  more  emphasis  on  cost  in  the 
decision  process.  With  the  launch  of  the  Better  Buying  Power  (BBP)  initiatives  and  the 
Weapon  Systems  Acquisition  Reform  Act  (WSARA),  affordability  has  been  mandated  as  a 
requirement  at  all  milestone  decision  points  of  program  development  (Carter,  2010a, 

2010b).  Designing  for  affordability  is  thus  imperative  to  early  phase  decision-making  in  the 
development  of  weapon  systems  and  programs. 

Affordability  has  become  a  design  requirement  due  to  multiple  instances  of  failure  in 
delivering  expected  technical  performance,  increased  costs  and  schedule  delays  beyond 
program  estimates,  and  the  altering  of  requirements  during  program  execution  (GAO,  2011). 
The  increasing  prominence  of  affordability  within  the  DoD  and  other  working  groups  has  led 
to  the  proposal  of  several  notable  definitions  for  the  term  affordability. 

(i)  The  2010  Carter  memorandum  defines  affordability  as  “conducting  a 
program  at  a  cost  constrained  by  the  maximum  resources  the  Department 
can  allocate  for  that  capability”  (Carter,  2010a,  2010b). 

(ii)  INCOSE  defines  affordability  as  “the  balance  of  system  performance,  cost 
and  schedule  constraints  over  the  system  life  while  satisfying  mission 
needs  in  concert  with  strategic  investment  and  organizational  needs” 
(INCOSE,  2012). 

(iii)  NDIA  defines  affordability  as  “the  practice  of  ensuring  program  success 
through  the  balancing  of  system  performance  (KPPs),  total  ownership  cost, 
and  schedule  constraints  while  satisfying  mission  needs  in  concert  with 
long-range  investment,  and  force  structure  plans  of  the  DOD”  (NDIA,  2011). 
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(iv)  The  Defense  Acquisition  Guidebook  defines  affordability  as  “the  degree  to 
which  the  life-cycle  cost  of  an  acquisition  program  is  in  consonance  with  the 
long-range  modernization,  force  structure  and  manpower  plans  of  the 
individual  DoD  Components,  as  well  as  for  the  Department  as  a  whole” 
(DoD,  2011). 

As  evidenced  by  this  set  of  definitions,  the  concept  of  affordability  not  only 
incorporates  cost  but  also  schedule,  performance,  lifecycle,  and  all  of  these  things  relative  to 
a  larger  set  of  possible  investments.  An  affordable  system  is  thus  cost  effective  on  its  own — 
and  relative  to  a  larger  system  investment  portfolio — in  delivering  value  to  the  customer  and 
relevant  stakeholders.  Affordability  is  enhanced  if  the  system  is  capable  of  satisfying 
possible  changing  mission  requirements  over  the  system  lifecycle.  Consequently,  a  system 
developed  without  consideration  for  affordability  is  one  that  has  been  designed  as  a  point 
solution  in  isolation,  to  meet  a  specific  need  at  a  specific  time,  possibly  requiring  the 
procurement  of  an  entirely  new  system  when  customer  needs  evolve  (Bobinis,  Haimowitz, 
Tuttle,  &  Garrison,  2012). 

Since  the  definitions  of  affordability  also  discuss  costs  relative  to  allocated  budgets, 
affordability  may  also  be  analyzed  at  various  levels  of  scope,  as  budgets  can  be  allocated  to 
systems,  programs,  and  even  portfolios  of  programs.  Budgets  and  development  timespans 
allocated  to  a  program  or  portfolio  may  be  partitioned  into  smaller  packages  in  many 
different  ways  among  its  constituent  systems  or  programs,  respectively. 

Affordability  at  a  higher  order  program  level  may  not  necessarily  equate  to 
affordability  at  a  lower  order  constituent  system  level  and  vice  versa.  Therefore,  different 
measures  may  have  to  be  applied  to  the  design  for  affordability  in  an  isolated  system, 
program,  or  portfolio,  as  well  as  for  the  intended  cascading  of  affordability  considerations 
from  higher  to  lower  levels  of  acquisition  management. 

Higher  order  levels  of  affordability  analysis  will  become  increasingly  important  in  the 
future,  as  they  can  stimulate  an  enterprise-driven  effort  to  perform  an  architectural 
transformation  of  traditional  engineering  design  methods  that  will  eventually  improve  the 
affordable,  full  lifecycle  operational  effectiveness  of  customer  solutions  (Bobinis  &  Herald, 
2012). 

Past  Failures 

The  consideration  of  affordability  as  a  requirement  during  early  phase  design  has 
become  necessary  in  recent  years  due  to  prominent  failures  in  system  and  overall  program 
delivery.  In  fiscal  year  (FY)  2012  (GAO,  2012),  nearly  half  of  the  DoD’s  96  largest  acquisition 
programs  were  failing  to  meet  the  “Nunn-McCurdy”  cost  growth  and  schedule  standards  that 
were  earlier  established  to  identify  troubled  programs  (Schwartz,  2010,  2013).  Despite 
active  reductions  in  weapon  unit  quantities  and  reduced  performance  expectations,  the  cost 
overruns  on  major  defense  acquisition  programs  have  grown  to  more  than  $300  billion  over 
original  program  estimates  (GAO,  2011). 

Notable  programs  that  experienced  cost  and  schedule  overruns  were  the  Army’s 
Comanche  armed  reconnaissance  helicopter,  the  Navy’s  DDG-1000  next-generation 
surface  combatant,  and  the  Air  Force’s  Transformational  Satellite  Communications  System 
(TSAT;  Cancian,  2010).  The  Comanche  program  commenced  in  1982,  but  increasing  unit 
costs  resulted  in  a  10-year  delay  in  schedule  and  its  eventual  cancellation  in  2004.  The  $6.9 
billion  initially  allocated  for  the  procurement  of  1 20  Comanche  helicopters  over  five  years 
could  have  been  directed  towards  upgrading  350  AH-64  attack  helicopters  to  deliver  greater 
warfighter  capability  but  was  instead  used  to  purchase  800  other  helicopters  (Cancian, 
2010). 
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Similarly,  the  DDG-1000  program  was  cancelled  in  2009  due  to  high  costs  and 
mission  limitations,  and  funds  were  instead  used  to  procure  additional  units  of  the  older 
DDG-51  model.  Unnecessary  expenditures  and  schedule  delays  might  have  been  averted  if 
the  Navy  initially  decided  to  purchase  13  units  of  the  DDG-51  class  for  its  $23  billion 
investment  in  only  three  DDG-1000  units.  TSAT  was  also  cancelled  in  2009  due  to  rising 
costs  and  schedule  slips.  The  Air  Force  might  have  used  the  $3.5  billion  initial  investment  in 
TSAT  to  purchase  seven  units  of  the  existing  Advanced  Extremely  High  Frequency  (AEHF) 
satellites  to  avoid  gaps  in  coverage  (Cancian,  2010). 

These  high-profile  failures  accentuate  the  urgent  need  to  reduce  cost  overruns  and 
schedule  delays,  as  well  as  the  need  to  consider  the  impact  of  switching  to  alternatives  later 
in  the  lifecycle,  in  the  design  of  both  current  and  future  defense  systems,  programs,  and 
portfolios.  With  recent  strategies  to  mandate  affordability  as  a  requirement,  establishing  the 
preliminary  design  choice  space  for  the  system,  program,  or  portfolio  of  interest  through  the 
generation  of  many  possible  alternatives  has  become  imperative  to  increase  insights  in  early 
phase  decision-making,  mitigate  the  risk  of  later  costly  changes,  and  maximize  the  value 
created  for  the  stakeholders  (Ross  &  Hastings,  2005). 

Affordability  and  Tradespaces 

The  Carter  memorandum  (Carter,  2010a)  stated  that  “the  ability  to  understand  and 
control  future  costs  from  a  program’s  inception  is  critical  to  achieving  affordability 
requirements.”  Since  cost  commitments  and  uncertainty  are  usually  at  their  highest  levels 
during  early  phase  design  (Blanchard  &  Fabrycky,  2006),  implementing  affordability 
tradeoffs  in  DoD  systems  and  programs  at  their  points  of  inception  can  actively  reduce 
future  cost  overruns  and  schedule  delays.  To  perform  affordability  tradeoffs  early  in  the 
lifecycle,  methods  for  systems  engineering  tradeoff  analysis  are  required  to  demonstrate 
changes  in  costs  as  major  decision  parameters  and  time  to  completion  are  varied. 

The  minimization  of  system  cost,  while  maintaining  or  increasing  system  capability 
across  changing  contexts  over  time,  motivates  the  construction  of  system  tradespaces  with 
consideration  of  temporality.  A  tradespace  is  the  space  spanned  by  possible  design 
alternatives  and  is  bounded  by  utility  and  cost.  As  the  alternatives  are  generated  by 
enumerating  design  variables,  expanding  the  tradespace  requires  a  “creative  recombination 
of  current  resources  or  systems  to  create  a  new  system,”  which  would  involve  generation  of 
either  new  design  variables  or  reconfigurations  of  existing  combinations  of  variables  (Ross, 
Hastings,  Warmkessel,  &  Diller,  2004). 

Leveraging  the  increasing  availability  of  computation  power,  tradespace  exploration 
is  the  utility-guided,  model-based  search  for  better  design  solutions  by  avoiding  premature 
fixation  on  point  designs  and  narrow  requirements  (Ross  et  al.,  2004).  This  allows  a  deeper 
and  more  holistic  consideration  of  capabilities  and  mission  utility  instead  of  being  locked  too 
early  into  requirements  and  key  performance  parameters  (Neches,  Carlini,  Graybill, 

Hummel,  &  McGrath,  2012). 

The  use  of  tradespaces  instead  of  simple  tradeoffs  of  several  point  designs  can  lead 
to  better  lifecycle  results  for  the  system  or  program  of  interest.  The  exploration  of 
tradespaces  also  enables  the  promulgation  of  ilities,  which  are  properties  that  often  manifest 
and  determine  value  in  a  system  after  it  is  put  into  use  and  which  have  ramifications  with 
respect  to  time  and  stakeholders  (de  Week,  Ross,  &  Rhodes,  2012). 

For  example,  flexibility  allows  for  leveraging  of  emergent  opportunities  and  mitigating 
risks  such  that  the  system  is  able  to  respond  to  changing  contexts  in  order  to  retain  or 
increase  value  delivery  to  system  stakeholders  over  time  (Viscito  &  Ross,  2009).  As  such, 
flexibility  has  become  a  desirable  quality  in  long-lived  systems.  Due  to  increasing  system 
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costs  and  operational  lifetimes  of  systems,  various  studies  have  already  been  conducted  to 
incorporate  flexibility  into  systems  during  the  conceptual  design  phase  (Saleh,  Mark,  & 
Jordan,  2009). 

For  the  purposes  of  this  paper,  affordability  will  be  a  concept  that  provides  explicit 
considerations  for  system  cost  and  schedule  constraints  in  value  delivery  and  sustainment 
to  stakeholders.  Designing  a  system  or  program  using  affordability  tradespaces  will  thus 
define  its  utility  or  performance  space  bounded  by  costs  and  time.  Affordability  studies  can 
thus  facilitate  the  attainment  of  better  system,  program,  or  portfolio  lifecycle  results  while 
meeting  budgetary  and  schedule  constraints. 

Applying  Epoch-Era  Analysis  to  Enable  Designing  for  Affordability 

The  analysis  of  affordability  tradespaces  can  become  a  multifaceted  process  when 
the  impacts  of  uncertainties  (including  risks)  inherent  in  alternative  futures  are  incorporated 
into  conducting  tradeoffs  during  acquisition.  To  effectively  evaluate  the  impact  of  dynamic 
variations  in  costs  with  tradeoffs  in  decision  parameters  and  time  to  completion,  an 
approach  called  Epoch-Era  Analysis  (EEA)  can  be  applied  to  enhance  the  design  for 
affordability  (Ross,  2006;  Ross  &  Rhodes,  2008). 

EEA  has  been  developed  to  consider  and  clarify  the  effects  of  changing  contexts  and 
needs  over  time  on  the  perceived  value  of  a  system  in  a  structured  manner  (Ross,  2006; 
Ross  &  Rhodes,  2008).  Instead  of  discretizing  the  system  lifecycle  according  to  traditional 
system  milestones,  EEA  discretizes  the  lifecycle  according  to  impactful  changes  in  the 
operating  environment,  stakeholders,  or  the  system  itself,  through  the  constructs  epochs 
and  eras. 

An  epoch  is  a  time  period  of  fixed  contexts  and  needs  under  which  the  system 
operates,  and  it  can  be  characterized  using  a  set  of  variables  that  define  any  factor,  such  as 
technology  level  and  supply  availability,  which  impacts  the  usage  and  value  of  the  system. 
An  ordered  sequence  of  epochs  constitutes  an  era  and  describes  the  potential  progression 
of  contexts  and  needs  over  time.  Any  futures  relevant  to  system  performance  or  costs  can 
be  described  through  assignments  to  the  available  epoch  variables,  providing  a  form  of 
computational  scenario  planning  (Roberts,  Richards,  Ross,  Rhodes,  &  Hastings,  2009). 

Figure  1  illustrates  a  notional  system  trajectory  across  an  ordered  sequence  of 
epochs  forming  an  era.  In  this  illustration,  the  impact  of  changing  contexts  can  be  seen  as  a 
downward  path  on  the  system  as  it  progresses  across  time.  Rising  expectations  are  also 
shown,  illustrating  how  the  perception  of  a  successful  system  can  be  dependent  not  only  on 
how  the  system  performs  within  a  context  but  also  how  that  performance  compares  to 
changing  expectations.  In  the  final  epoch  of  the  illustrated  era,  the  system  must  change  in 
order  to  meet  expectations.  In  this  way,  EEA  can  structure  consideration  of  changing 
contexts  and  needs  on  system  success  and  suggest  strategies  for  how  to  sustain  value  in 
both  the  short  run  and  the  long  run. 
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Figure  1.  Partitioning  Short  Run  and  Long  Run  Into  Epochs  and  Eras 

(Ross  &  Rhodes,  2008) 

EEA  can  be  used  with  dynamic  Multi-Attribute  Tradespace  Exploration  (MATE),  a 
conceptual  design  method  that  generates  large  numbers  of  designs  through  combinations  of 
nonlinear  functions  of  their  performance  attributes  and  compares  their  costs  and  utilities 
(Ross  et  al.,  2004).  Enumeration  and  evaluation  of  many  alternative  designs  allow  for  a 
more  complete  exploration  of  a  larger  design  tradespace.  Evaluation  of  a  single  point  design 
in  which  time-dependent  performance  variables  are  present  can  also  be  performed. 

Therefore,  the  application  of  EEA  to  designing  for  affordability  in  a  system,  program, 
or  portfolio  can  allow  analysis  of  value  delivery  for  single  or  multiple  point  designs  across 
multiple  epochs  and  multiple  eras.  System  engineers  can  thus  contribute  to  realizing  better 
buying  power  by  examining  affordable  systems  previously  overlooked  or  discarded  (e.g., 
more  affordable  solutions  may  emerge  from  previously  neglected  regions  of  the 
tradespace). 

Multi-Attribute  Expense  in  Designing  for  Affordability 

Designing  for  affordability  is  not  only  concerned  with  the  monetary  lifecycle  cost  of  a 
system.  While  many  definitions  of  affordability  exist,  there  is  general  consensus  that  any 
evaluation  of  affordability  must  include  a  system’s  “schedule”  of  development  and  its 
responsiveness  to  emerging  needs  (Mallory,  2009;  Herald,  2011;  INCOSE,  2012). 

However,  such  temporal  considerations  are  often  difficult  or  impossible  to  represent  in 
dollars.  Non-monetary  measures  beyond  traditional  forms  like  lifecycle  cost  are  thus 
required.  An  additional  concern  is  that  dollars  for  a  system  are  often  allocated  from  different 
budgets — for  example,  development  versus  operations.  These  different  “colors”  of  money 
may  be  allocated  (and  spent)  with  differing  degrees  of  ease.  Analysis  without  aggregating 
these  different  types  of  dollar  budgets  may  provide  additional  insights  that  would  otherwise 
be  lost  if  dollars  were  aggregated  into  a  single  monetary  measure. 

A  possible  measure  capable  of  keeping  track  of  both  monetary  and  non-monetary 
considerations,  as  well  as  keeping  different  colors  of  money  separate,  is  the  Multi-Attribute 
Expense  (MAE)  function,  which  has  previously  been  used  in  a  satellite  system  design  case 
study  as  an  independent  variable  in  tradespace  exploration  to  capture  both  a  system’s 
development  time  and  initial  operating  costs  (Diller,  2002).  MAE  is  formulated  similarly  to  a 
Multi-Attribute  Utility  (MAU)  function  (Keeney  &  Raiffa,  1993).  Expense  refers  to  aspects  of 
the  system  design  and  development  that  the  designer  wants  to  keep  at  low  levels,  a  concept 
akin  to  the  notion  of  negative  utility.  Expense  is  principally  focused  on  “what  goes  into  a 
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system,”  in  contrast  to  utility,  which  is  focused  on  “what  comes  out  of  a  system.”  Typically 
quantified  on  a  zero  to  one  scale,  where  an  expense  level  of  one  denotes  complete 
dissatisfaction  and  an  expense  level  of  zero  denotes  minimal  dissatisfaction.  As  such,  a 
stakeholder  typically  demands  maximal  utility  and  minimal  expense  in  an  ideal  design 
(Nickel,  2010). 

An  MAE  function  requires  careful  construction  through  stakeholder  interviews  to  elicit 
informed  responses  and  aggregate  preferences  to  capture  articulated  value.  Because  MAE 
is  a  dimensionless,  non-ratio  scale  metric,  an  entity  with  twice  the  MAE  number  over  another 
does  not  imply  that  it  is  twice  as  expensive  in  terms  of  monetary  value. 

Since  temporal  elements  like  schedule  constraints  and  time-to-build  have  extensive 
leverage  on  the  different  colors  of  money,  the  MAE  can  be  extended  to  affordability 
applications  in  federal  acquisition  processes.  Instead  of  comparing  monetary  costs  against 
utility,  EEA  can  be  modified  to  compare  MAE  against  MAU  in  order  to  perform  affordability- 
driven  analysis  that  captures  the  elements  of  both  time  and  costs. 

A  method  that  leverages  the  EEA  approach  and  MAE  metric  can  allow  for  the 
effective  comparison  of  benefits  and  costs  across  a  range  of  alternative  futures.  Also,  this 
method  may  transform  traditional  engineering  practices  in  acquisition  management  if  it  is 
able  to  account  for  system  changes  due  to  shifts  and  perturbations,  manage  lifecycle 
differences  between  subsystem  or  subprogram  components,  evaluate  feedback,  and  be 
adaptive  to  evolving  system  behaviors  (Bobinis  et  al. ,  2012).  Since  affordability  is  a  concept 
evaluated  over  time,  such  a  method  can  provide  structured  options  for  improvement  to 
enable  enhanced  design  for  affordability. 

Proposed  Method  Based  Upon  Epoch-Era  Analysis 

A  new  method  that  leverages  the  EEA  approach  and  MAE  metric  is  proposed  to  help 
enable  the  design  of  affordable  systems,  allowing  for  the  structured  evaluation  of  design 
alternatives  across  many  alternative  futures  and  helping  to  ensure  that  a  potential  design’s 
cost  is  acceptable  across  the  entire  lifecycle.  The  proposed  method  is  inspired  by  the 
Responsive  Systems  Comparison  (RSC)  method,  which  was  developed  earlier  to  support 
designing  for  changeability  (Ross  et  al.,  2008).  RSC  is  a  prescriptive  operationalization  of 
MATE  and  EEA.  RSC  has  been  previously  applied  to  the  design  of  a  satellite  radar  system 
(Ross,  McManus,  Rhodes,  Hastings,  &  Long,  2009). 

RSC  is  designed  to  “guide  the  ...  practitioner  through  the  steps  of  determining  how  a 
system  will  deliver  value,  brainstorming  solution  concepts,  identifying  variances  in  contexts 
and  needs  (epochs)  that  may  alter  the  perceived  value  delivered  by  the  system  concepts, 
evaluating  key  system  tradeoffs  across  varying  epochs  (eras)  to  be  encountered  by  the 
system,  and  lastly  developing  strategies  for  how  a  designer  might  develop  and  transition  a 
particular  system  concept  through  and  in  response  to  these  varying  epochs”  (Ross  et  al., 
2008).  It  is  hypothesized  that  through  modifying  several  original  processes  in  RSC, 
incorporating  recent  refinements  to  EEA,  and  utilizing  MAE  to  better  capture  the  diversity  of 
expenditures  on  a  given  system,  the  proposed  method  can  more  effectively  address  the 
time  and  resource-centric  approach  of  designing  for  affordability. 

Overview  of  Proposed  Method 

The  overall  structure  of  the  proposed  method  consists  of  nine  processes,  which  are 
grouped  into  three  distinct  parts:  information  gathering  (Processes  1  through  3),  alternatives 
evaluation  (Process  4),  and  alternatives  analysis  (Processes  5  through  9).  A  graphical 
representation  of  the  method  is  shown  in  Figure  2.  The  information-gathering  portion, 
Processes  1  through  3,  consists  of  defining  the  context  and  problem  statement, 
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stakeholders  and  respective  needs,  and  contextual  variables.  The  alternatives  analysis 
portion,  Processes  5  through  9,  compares  the  dynamic  properties  of  potential  designs 
across  the  potential  futures  that  the  system  may  encounter.  These  two  main  portions  of  the 
proposed  method  are  bridged  by  Process  4  (Design-Epoch  Tradespaces  Evaluation),  which 
can  provide  feedback  to  decision-makers  and  stakeholders,  creating  an  opportunity  to  revisit 
the  information-gathering  processes.  Process  4  also  provides  cursory  analysis  of  potential 
designs  in  preparation  for  the  more  in-depth  alternatives  analysis  in  the  second  half  of  the 
method. 


Design  Analysis 


5.  Single  Epoch 

6.  Multi-Epoch 

7.  Era 

8.  Single-Era 

9.  Multi-Era 

Analyses 

Analysis 

Construction 

Analyses 

Analysis 

Figure  2.  A  Graphical  Overview  of  the  Gather-Evaluate-Analyze  Structure  of  the  Method 

The  processes  of  the  proposed  method,  with  brief  descriptions  of  the  activities 
involved,  are  as  follows,  with  modifications  to  the  prior  RSC  method  emphasized  (in  italics): 

Process  1 :  Value-Driving  Context  Definition 

The  first  process  of  the  proposed  method  involves  development  of  the  basic  problem 
statement.  The  stakeholders  are  identified,  relevant  exogenous  uncertainties  are  elicited, 
and  an  initial  value  proposition  is  formed.  The  resources  available  to  each  stakeholder  are 
examined  along  with  the  associated  uncertainties. 

Process  2:  Value-Driven  Design  Formulation 

The  second  process  begins  by  defining  the  needs  statements  for  all  stakeholders, 
which  become  the  attributes  of  system  performance,  along  with  utility  functions  describing 
each  stakeholder’s  preference  for  each  attribute.  The  stakeholder  resources  statements  are 
also  elicited  (with  corresponding  expense  functions),  which  then  become  the  attributes  of 
the  system’s  expense  function.  The  system  solution  concepts  are  proposed  from  past 
concepts  or  expert  opinions.  These  concepts  are  decomposed  into  design  variables  of  the 
system. 
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Process  3:  Epoch  Characterization 

In  this  process,  the  key  contextual  uncertainties  are  parameterized  as  epoch 
variables,  and  possible  future  contexts  are  identified.  Uncertainties  in  stakeholder  needs  are 
elicited.  Uncertainties  in  resource  supply  and  availability  are  also  identified,  along  with 
changes  to  stakeholder  preferences  on  resource  usage. 

Process  4:  Design-Epoch  Tradespaces  Evaluation 

This  process  utilizes  modeling  and  simulation  to  map  the  design  and  epoch  variables 
to  system  performance  attributes  and  expense  attributes.  Stakeholders’  utility  and  expense 
functions  are  then  used  to  generate  the  MAU  and  the  MAE  for  each  design,  within  each 
epoch. 

Process  5:  Single  Epoch  Analyses 

This  process  includes  the  analysis  of  MAU  and  MAE  of  alternatives  within  particular 
epochs,  including  designs  graphically  compared  on  an  MAU  versus  MAE  scatterplot  for  any 
given  epoch  (time  period  of  fixed  operating  context  and  stakeholder  needs).  Within-epoch 
metrics,  such  as  yield,  give  an  indication  of  the  difficulty  of  a  particular  context  and  needs 
set  for  considered  designs. 

Process  6:  Multi-Epoch  Analysis 

After  completing  the  traditional  tradespace  exploration  activities  of  Process  5,  in 
which  the  practitioner  compares  potential  designs  within  a  particular  epoch,  metrics  are 
derived  from  measuring  design  properties  across  multiple  (or  all)  epochs  to  give  insight  into 
the  impact  of  uncertainties  on  potential  designs,  including  the  evaluation  of  short  run  passive 
and  active  strategies  for  affordability  (i.e. ,  efficient  MAU  at  MAE).  In  addition,  resource 
usage  can  be  analyzed  to  identify  designs  that  are  robust  to  the  factors  identified  in  Process 
3  (e.g.,  decreasing  budgets  or  labor  availability). 

Process  7:  Era  Construction 

This  process  constructs  multiple  sequences  of  various  fixed  duration  epochs 
together  to  create  alternative  eras,  which  are  long-term  descriptions  of  possible  futures  for 
the  system,  its  context,  and  stakeholder  needs.  This  process  can  be  performed  with  the  aid 
of  expert  opinion,  probabilistic  models  (e.g.,  Monte  Carlo  or  Markov  models),  and  scenarios 
of  interest  to  stakeholders. 

Process  8:  Single-Era  Analyses 

This  process  examines  the  time-dependent  effects  of  an  unfolding  sequence  of 
future  epochs  created  in  Process  7.  By  examining  a  particular  series  of  epochs  for  a  given 
length  of  time,  decision-makers  can  identify  potential  strengths  and  weaknesses  of  a  design 
and  better  understand  the  potential  impact  of  path-dependent,  long-run  strategies  for 
affordability. 

Process  9:  Multi-Era  Analysis 

This  process  extends  Process  8  by  evaluating  the  dynamic  properties  of  a  system 
across  many  possible  future  eras,  identifying  patterns  of  strategies  that  enable  affordability 
across  uncertain  long-run  scenarios. 

In  the  remainder  of  the  paper,  the  first  three  processes  are  described  in  more  detail 
for  the  purposes  of  the  demonstration  case  (as  of  the  date  of  this  paper),  with  the  modeling 
and  simulation  and  the  analysis  processes  to  be  applied  later  in  the  ongoing  effort. 
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Offshore  Patrol  Cutter  Acquisition  Demonstration  Case 

The  case  chosen  for  an  initial  demonstration  of  the  proposed  method  is  drawn  from 
Schofield  (2010),  who  described  an  $8  billion  Coast  Guard  Offshore  Patrol  Cutter  (OPC) 
acquisition  program  for  over  20  ships,  each  with  a  service  life  of  around  30  years.  For  this 
paper,  only  the  acquisition  of  one  ship  will  be  considered,  and  the  alternatives  will  be  limited 
to  a  few  point  designs  rather  than  an  exhaustive  tradespace  of  alternative  designs.  A  brief 
description  of  the  project  is  given.  Ongoing  work  will  extend  the  analysis  to  the  program 
level  to  examine  measures  of  affordability  on  multi-year  and  multi-unit  acquisitions. 

The  OPC  operates  in  a  variety  of  areas  to  perform  many  different  missions,  including 
ports,  waterways,  and  coastal  security  (PWCS),  search  and  rescue  (SAR),  drug  interdiction 
(DRUG),  migrant  interdiction  (AMIO),  living  marine  resources  (LMR),  other  law  enforcement 
(OLE),  and  defense  readiness  (DR;  Fabling,  2010).  These  mission  areas  include 
autonomous  operations  as  well  as  cooperative  missions  with  other  vessels,  requiring 
endurance  and  maneuverability,  respectively. 

Process  1 :  OPC  Value-Driving  Context  Definition 

The  value-driving  context  for  the  OPC  is  made  up  of  the  value  propositions  as  well 
as  the  key  stakeholders  involved  in  decision-making  and  funding.  The  basic  stakeholder 
relationships  present  for  the  OPC  system  are  depicted  in  Figure  3.  As  shown  in  the  figure, 
the  internal  stakeholders  are  the  entities  between  which  the  primary  exchanges  of  value 
occur. 


Figure  3.  A  Graphical  Deptiction  of  the  Stakeholders  and  Their  Relationships  in 

the  OPC  System 

(Schofield,  2010) 
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Schofield  (2010)  defined  the  value  propositions  for  each  stakeholder  as  follows: 

Project  Office:  Provide  a  new  cutter  fleet  meeting  operational  requirements 
within  a  defined  budget  level  and  delivery  to  coincide  with  decommissioning 
of  current  WMEC  fleet. 

Sponsor:  Develop  operational  requirements  that  meet  the  mission  needs  of 
the  Coast  Guard  and  Coast  Guard  user  requirements. 

Technical  Authorities:  Ensure  new  developed  system  meets  legacy, 
external  constraints,  and  design  standards  with  technologies  that  maximize 
capability  within  established  risk  requirements,  (p.  82) 

It  is  clear  from  the  value  propositions  that  concern  for  resource  usage  is  not 
consistent  across  stakeholders;  as  one  might  expect,  each  stakeholder  has  different 
expectations  and  goals  with  regard  to  resources  involved  in  the  project.  The  Project  Office 
specifically  addresses  two  standard  resources:  budget  (“defined  budget  level”)  and  schedule 
(“delivery  to  coincide  with  ...”).  The  Sponsor  appears  to  be  primarily  concerned  with  the 
mission  needs  and  user  requirements  of  the  organization,  and  resource  usage  is  not  of 
primary  concern.  The  Technical  Authorities’  value  statement  includes  the  aspect  of 
technological  resources  that  enable  core  capabilities.  In  the  second  process  of  the  proposed 
method,  interviews  of  each  stakeholder  will  better  reveal  their  preferences  on  the  usage  of 
the  resources  with  which  they  are  concerned. 

Process  2:  Value-Driven  Design  Formulation 

The  second  process  builds  upon  the  initial  system  context  definition  by  first 
proposing  the  system  design  concept  and  then  eliciting  the  performance  attributes  desired 
by  the  stakeholders.  The  design  concepts  are  then  partitioned  into  potential  design  variables 
for  the  proposed  system.  To  better  identify  the  key  design  drivers,  the  relationships  of 
design  variables  to  performance  attributes  are  then  assessed  qualitatively  by  the  values 
“none,”  “low,”  “medium,”  or  “high”  impact,  using  a  Design-Value  Matrix  (DVM)  as  a  visual  aid 
for  this  activity.  Schofield  (2010)  decomposed  the  value  propositions  generated  in  Process  1 
to  infer  the  performance  attributes  and  then  map  them  to  the  design  variables  of  the  system. 

For  the  present  study,  a  new  matrix  is  created  to  map  the  impact  of  design  variables 
to  the  resource  expenditures  of  the  system,  qualitatively  assessing  each  design  variable’s 
impact  on  each  expense  attribute.  In  this  way,  an  alternative  DVM  can  be  generated, 
focused  on  expense  attributes,  and  is  shown  in  Figure  4  with  purely  notional  data.  By 
summing  the  rows  and  columns,  the  practitioner  can  quickly  determine  which  design 
variables  have  the  most  impact  on  general  resource  usage  (in  the  notional  example,  the 
length  and  propulsion  type  are  the  most  impactful),  as  well  as  which  resources  are  more 
sensitive  to  the  present  design  choices  (again,  from  the  notional  data,  the  Operations  Cost 
is  the  most  sensitive,  followed  by  Blue  Money  and  Acquisition  Cost).  Generating  an 
enhanced  DVM,  with  both  utility  attributes  and  expense  attributes,  provides  an  expanded 
cost  and  benefit  perspective  on  the  ramifications  of  various  design  decisions. 
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Expense-Focused  DVM  (Notional  Values) 


Key  Internal  Stakeholder:  Sponsor 


Expense 

Attributes 

Design  Variables 

Length  (ft) 

Power  (hp) 

Propulsion  Type  (Diesel, 

CODOG,  CODAG,  Turbo) 

Antennae  Space  (cub.  Ft) 

Crew  site  (ft) 
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•  •• 
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3 

1 
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25 
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31 

11 

25 

17 

12 

... 

Figure  4.  A  Design-Value  Matrix  Reflecting  the  Notional  Impact  of  Design  Variables  on 

System  Expense  Attributes 

Process  3:  Epoch  Characterization 

After  identification  of  the  design  variables,  performance  and  expense  attributes,  and 
their  corresponding  relationships,  the  internal  and  external  uncertainties  are  added  into  the 
analysis.  The  prior  study  (Schofield,  2010)  listed  the  external  uncertainties  (in  the  associated 
categories)  related  to  the  OPC  as  follows: 

Technology:  VUAV  integration;  major  C4ISR  system  upgrade;  and  new  and 
more  capable  (size,  range,  personnel  carried)  small  boats. 

Policy:  Marine  engine  emission  reductions;  reduced  copper  content  from 
shipboard  systems  (sea  water  systems);  increased  intelligence  gathering  into 
government-wide  system. 

Budget:  Loss  of  acquisition  budget  prior  to  IOC;  increase  in  operational 
funding  for  increased  operational  usage. 

Systems  of  Systems:  Deploying  with  National  Security  Cutters;  new  cutter- 
deployed  helicopters. 

Missions:  Support  of  arctic  region  for  fisheries;  adding  environmental 
cleanup  response  capability;  more  frequent  international  presence  particularly 
for  peacekeeping  missions,  (p.  93) 

Epoch  variables  are  generated  from  these  uncertainties  by  determining  the  primary 
source  of  the  possible  changes  in  operating  context.  For  instance,  Schofield  (2010)  used  the 
marine  engine  emission  reductions  uncertainty  in  the  Policy  category  to  generate  the 
“Engine  Emissions  Rating”  epoch  variable,  which  has  an  integer  value  range  from  2  to  4. 
Once  each  epoch  variable  is  created,  the  impact  of  the  epoch  variables  on  each  of  the 
design  variables,  performance  attributes,  and  resource  attributes  can  then  be  depicted  with 
an  Epoch  Descriptor  Impact  Matrix,  similar  to  the  DVM  in  Process  2.  An  example  Epoch 
Descriptor  Impact  Matrix  with  notional  values  is  shown  in  Figure  5. 
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Epoch  Descriptor  Impact  Matrix 
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Figure  5.  A  Matrix  Reflecting  the  Notional  Impact  of  Epoch  Variables  on  Design 
Variables,  Utility  Attributes,  and  Expense  Attributes 

Similar  conclusions  can  be  drawn  as  in  Process  2;  for  example,  it  is  clear  from  the 
sums  of  rows  in  Design  Variables  that  Power  is  the  variable  most  impacted  in  general  by  all 
uncertainties.  Likewise,  Range  is  the  performance  attribute  most  impacted  by  the 
uncertainties,  and  Blue  Money  is  the  most  impacted  expense  attribute.  Conversely,  the 
Small  Boat  Size  epoch  variable  (with  Engine  Emissions  not  far  behind)  is  the  most  impactful 
on  all  design  variables,  the  SCIF  Size  is  most  impactful  on  the  utility  attributes,  and  the 
VUAV  is  most  impactful  on  the  expense  attributes.  Gaining  an  understanding  of  these 
relationships  early  in  the  design  process  allows  the  practitioner  to  begin  considering  how  a 
design  should  be  oriented  to  cope  with  uncertainties  as  well  as  to  keep  in  mind  those 
contexts  which  are  especially  detrimental  to  the  utility  or  expense  of  the  system,  whether 
directly  or  through  opportunity  costs. 

Next  Steps:  Processes  4-9 

The  study  is  continuing  beyond  this  paper  with  the  application  of  the  second  half  of 
the  proposed  method.  Process  4,  the  Design-Epoch  Tradespaces  Evaluation,  will  use  the 
information  generated  thus  far  to  calculate  the  expense  measurements  for  each  design, 
allowing  a  partial  tradespace  to  be  shown  on  a  standard  utility-versus-expense  scatterplot. 
Preliminary  results  can  be  shown  to  stakeholders  and  decision-makers,  allowing  feedback  to 
Processes  2  and  3,  if  necessary,  to  update  the  design  variables  and  epoch  variables  under 
consideration  (see  Figure  2).  Upon  completion  of  Process  4  and  any  necessary  iteration, 
Process  5  (Single-Epoch  Analyses)  will  then  begin  to  look  at  the  designs’  relative  resource 
utilization  in  different  epochs,  allowing  the  practitioner  to  begin  understanding  the  dynamic 
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expense  properties  of  each  alternative  under  consideration  as  well  as  to  pick  relevant  point 
futures  in  which  to  compare  alternative  designs. 

The  application  will  continue  with  the  Multi-Epoch  Analysis  of  Process  6,  measuring 
the  designs’  expenses  across  all  relevant  epochs.  Established  metrics  such  as  the  Fuzzy 
Pareto  Number,  Filtered  Outdegree,  and  others  will  be  calculated  to  help  identify  designs 
that  are  insensitive  to  the  impact  of  uncertain  operating  environments  and  missions 
(Fitzgerald  &  Ross,  2012a).  Multi-epoch  affordability  metrics  will  also  be  introduced  to  help 
identify  and  rank  designs  based  on  resource  usage,  including  those  which  best  adapt  to 
decreasing  budgets,  those  which  do  not  vary  widely  in  resource  usage,  and  those  which  can 
best  capitalize  on  increasing  technology  and  spending  levels. 

In  addition  to  observing  design  properties  across  many  alternative  point  futures,  it  is 
also  informative  to  analyze  design  properties  through  the  long  run  using  an  ordered 
sequence  of  different  epochs  (i.e.,  an  era).  A  particular  era  can  be  created  by  combining 
epochs  through  expert  opinion,  random  generation,  and  other  means.  During  Process  7  (Era 
Construction),  one  or  more  of  these  methods  will  be  used  to  generate  and  name  possible 
eras  for  the  OPC.  In  Process  8  (Single-Era  Analyses),  metrics  of  the  alternative  designs  will 
then  be  compared  for  a  given  era  as  an  indication  of  design  resource  usage  in  one  possible 
long-run  future,  potentially  revealing  the  path-dependent  sensitivity  of  a  particular  design 
(Fitzgerald  &  Ross,  2012b).  This  analysis  will  be  broadened  to  identify  patterns  across 
multiple  possible  long-run  epoch  sequences  in  Process  9,  Multi-Era  Analysis,  wherein 
further  insights  into  the  long-term  resource  behavior  of  the  OPC  under  many  different 
scenarios  will  be  gained.  These  insights  will  be  aided  through  the  application  of  existing  and 
proposed  metrics  to  compare  properties  such  as  the  stability  of  operating  costs,  stability  of 
manpower  requirements,  and  adaptation  to  budget  variation. 

Discussion 

Designing  for  affordability  throughout  the  acquisition  process,  but  especially  in  the 
early  phases,  can  expedite  significant  reductions  in  cost  and  risk  and  enable  the  value 
delivery  of  either  effective  systems  or  programs  within  economically  frugal  and  risk-averse 
environments.  By  applying  the  proposed  method  to  a  demonstration  study,  we  intend  to 
illustrate  how  affordable  solutions  within  fluid  tradespaces  can  be  identified  in  a  systematic 
and  informed  manner,  accounting  for  changing  sets  of  mission  requirements,  operating 
contexts,  and  available  budgets. 

As  the  preceding  demonstration  begins  to  illustrate  the  application  of  affordability 
considerations  to  a  single  acquisition  program,  the  same  method  can  also  be  applied  at  the 
levels  of  systems,  programs,  and  portfolios  of  programs.  Introducing  different  levels  of  scope 
to  affordability  studies  necessitates  clear  distinctions  among  systems,  programs,  and 
portfolios  in  future  work. 

For  the  purposes  of  the  current  discussion,  the  distinction  between  system,  project, 
and  program  is  now  described.  A  system  is  typically  defined  to  be  a  combination  of 
interacting  elements  organized  to  achieve  one  or  more  stated  purposes  (INCOSE,  2012), 
whereas  a  project  can  be  defined  as  the  enveloping  process  that  encompasses  the 
socioeconomic  and  technical  considerations  in  delivering  the  system.  A  program  can  be 
defined  as  a  group  of  related  and  interdependent  projects  managed  together  to  obtain 
specific  benefits  and  controls  that  would  likely  not  occur  if  these  projects  were  managed 
individually  (KLR,  2008). 

Program-level  affordability  can  be  achieved  through  either  a  top-down  or  bottom-up 
approach.  A  top-down  approach  entails  the  application  of  affordability  considerations  at  the 
program  level  such  that  its  effects  potentially  cascade  down  to  its  constituent  systems.  A 
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bottom-up  approach  conversely  demands  the  aggregation  of  system-level  affordability  for 
each  constituent  system  in  order  to  establish  program-level  affordability.  These  two 
approaches  may  not  yield  the  same  results,  and  it  is  an  avenue  worth  exploring  to  determine 
the  more  effective  or  viable  option. 

Another  paradigm  for  exploration  is  that  of  portfolio-level  affordability.  A  portfolio  can 
be  defined  as  a  collection  of  projects  or  programs  grouped  together  to  facilitate  the  effective 
management  of  efforts  to  meet  strategic  business  objectives  (KLR,  2008).  These  projects  or 
programs  are  not  necessarily  interdependent  or  directly  related.  Portfolio-level  affordability 
analysis  may  involve  applying  affordability  considerations  across  multiple  projects, 
programs,  and  possibly  even  portfolios.  Given  that  the  DoD  has  been  evaluating  the 
expenditures  for  both  defense  acquisition  programs  and  portfolios,  a  portfolio-level 
affordability  study  can  provide  overarching  guidance  to  architecting  entire  defense 
capabilities  within  realistic  bounds  of  cost  and  time.  Similarly,  top-down  and  bottom-up 
approaches  can  also  be  taken  to  achieve  portfolio-level  affordability. 

Quantitative  measures  of  cost,  schedule,  and  performance  can  also  be  derived  at 
system,  program,  and  portfolio  levels  to  serve  as  vital  health  indicators  at  different  tiers  of 
acquisition  processes.  By  assessing  the  performance  of  an  individual  system  or  program  as 
well  as  entire  defense  portfolios,  the  return  on  investment  that  the  current  defense 
acquisition  system  delivers  to  stakeholders  can  be  accurately  measured  and  analyzed. 
Systems,  programs,  and  portfolios  can  then  be  individually  or  collectively  calibrated  based 
on  a  multitude  of  affordability  indicators  in  order  to  fulfill  evolving  cost  and  schedule 
constraints. 

As  a  starting  point,  affordability  requirements  have  already  been  mandated  at  current 
program  milestone  reviews  and  at  the  inception  of  new  programs  in  consonance  with  the 
Carter  memorandum.  Designing  for  affordability  can  be  extended  to  all  levels  in  acquisition 
management  in  order  to  ensure  that  value  delivery  in  systems,  programs,  and  portfolios  can 
be  sustained.  The  method  described  in  this  paper  could  potentially  be  used  to  gain  insights 
at  each  of  these  levels  of  acquisition  management. 

Another  avenue  for  study  is  the  concurrent  application  of  affordability  with  other  ility 
considerations.  Research  on  the  tradeoffs  for  changeability  (Fitzgerald  &  Ross,  2012b), 
survivability  (Richards,  Ross,  Hastings,  &  Rhodes,  2009),  and  evolvability  (Beesemyer, 
Fulcoly,  Ross,  &  Rhodes,  2011)  in  previous  case  studies  may  possibly  be  repeated  with 
affordability  considerations,  and  results  may  yield  “affordably  changeable,”  “affordably 
survivable,”  or  “affordably  evolvable”  systems  that  were  previously  overlooked  or  discarded. 
For  such  change-related  ilities,  existing  methods  such  as  the  Epoch  Syncopation 
Framework  (ESF)  can  also  be  utilized  to  account  for  cost,  performance,  schedule,  and 
uncertainty  factors  regarding  experienced  epoch  shifts  in  the  analysis  of  design  tradespaces 
(Fulcoly,  Ross,  &  Rhodes,  2012).  This  can  promote  the  development  of  rules  and  strategies 
to  execute  change  mechanisms  with  explicit  considerations  for  cost  and  time  across  a 
system  lifespan. 

Conclusion 

Current  strategies  to  mandate  designing  for  affordability  as  a  requirement  in 
acquisition  management  are  still  at  their  infancy  stages.  A  particularly  challenging  aspect  of 
designing  for  affordability  is  to  make  decisions  not  only  for  today’s  mission  and  operating 
context  but  also  for  alternative  futures  where  budgets  and  missions  may  change  and 
contexts  may  shift.  We  propose  a  method  that  can  enable  practitioners  to  design  for 
affordability  in  a  more  effective  and  comprehensive  manner,  given  this  challenge  of  a 
dynamic  world.  By  leveraging  the  EEA  approach  and  the  MAE  metric,  tradespaces 
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containing  many  possible  design  alternatives  can  be  explored  with  stronger  considerations 
of  aggregated  costs,  time,  and  performance  in  order  to  focus  specifically  on  evaluating 
affordability.  This  will  facilitate  the  conduct  of  affordability  tradeoffs  in  dynamic  operating 
environments  with  many  possible  alternative  futures  and  eliminate  design  restrictions  to  only 
single  point  futures.  The  viability  of  this  method  has  been  preliminarily  demonstrated  with 
the  OPC  acquisition  case  study.  It  will  be  further  elaborated  in  the  course  of  ongoing 
research.  Inspired  by  the  existing  Responsive  Systems  Comparison  method,  this  method 
places  increased  emphasis  on  tradeoffs  important  to  managing  changes  in  available  and 
expended  resources  over  time.  It  is  anticipated  that  the  method  could  be  applied  to 
ascending  levels  of  acquisition  considerations,  from  systems  to  programs  to  portfolios. 
Program-level  affordability  analysis  might  consider  the  acquisition  management  of  the  entire 
OPC  acquisition  program  for  over  20  ships,  while  portfolio-level  affordability  analysis  might 
entail  the  concurrent  acquisition  management  of  the  OPC  program  with  other  related  or 
unrelated  naval  programs.  With  this  method,  the  concept  of  affordability  may  be  considered 
simultaneously  with  other  ilities,  providing  synergistic  insights  from  other  existing  methods 
such  as  the  ESF  to  enhance  the  design  of  affordable  systems.  Based  on  early  results,  the 
method  appears  to  show  promise  as  an  enabler  for  better  design  for  affordability  in  systems, 
programs,  and  portfolios  in  the  future. 
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Abstract 

The  efficacy  of  defense  acquisition  is  highly  dependent  on  acquisition  workforce  (AWF) 
quality,  but  assessing  such  quality  remains  a  major  challenge,  particularly  given  the 
knowledge-intensive  and  dynamic  nature  of  acquisition  organizations  and  processes.  Hence, 
it  is  difficult  to  gauge — much  less  predict — the  impact  of  leadership  interventions  in  terms  of 
policy,  process,  regulation,  organization,  education,  training,  or  like  approaches.  Building 
upon  the  development  and  application  of  Knowledge  Flow  Theory  over  the  past  couple  of 
decades,  we  have  developed  a  state-of-the-art  approach  that  enables  us  to  analyze, 
visualize,  and  measure  dynamic  knowledge  and  performance.  The  main  idea  is  to  apply  this 
approach  inwardly  to  measure  the  dynamic  knowledge  and  performance  of  acquisition 
processes  (e.g.,  within  contracting  and  project  management  organizations),  but  we  also  look 
outwardly  (e.g.,  at  warfare  processes  at  the  tactical  edges  of  military  combat  organizations)  to 
conceptualize  an  operational  proxy  for  acquisition  workforce  quality:  end  customer 
performance.  This  proxy  offers  its  best  potential  to  complement,  not  replace,  other  metrics  in 
use,  development,  and  conceptualization  today,  but  it  arguably  concentrates  on  one  of  the 
most  important  AWF  quality  determinants:  how  acquired  systems  affect  operational 
performance. 

Introduction 

Acquisition  is  big  business.  The  DoD  alone  routinely  executes  12-figure  budgets  for 
research,  development,  procurement,  and  support  of  weapon  systems  and  other  military 
products  and  services  (Dillard  &  Nissen,  2005).  Acquisition  is  also  a  knowledge-intensive 
business.  In  addition  to  myriad  laws  governing  federal  acquisition  in  the  U.S.,  a  plethora  of 
rules  and  regulations  specify — often  in  great  detail — how  to  accomplish  the  planning,  review, 
execution,  and  oversight  of  defense  acquisition  programs,  large  and  small,  sole-source  and 
competitive,  military  and  commercial  (Dillard,  2003). 

As  a  result  in  part — and  due  to  high  complexity,  multiple  stakeholders,  goal 
incongruence,  open  process  execution,  and  large  pecuniary  rewards  for  some  participants — 
acquisition  has  been  a  problematic  business  too.  Seemingly  every  decade,  acquisition 
problems  must  be  addressed  by  another  Blue  Ribbon  panel  and  reformed  yet  again.  The 
Better  Buying  Power  Initiatives  (BBPI),  as  a  recent  instance,  mandate  efficiency  and 
productivity  improvements  in  five  acquisition  business  areas:  (1)  affordability  and  cost 
growth,  (2)  productivity  and  innovation  in  industry,  (3)  competition,  (4)  tradecraft  in  services 
acquisition,  and  (5)  non-productive  processes  and  bureaucracy  (Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)],  2010). 
These  initiatives  focus  principally  on  incentives  for  and  interactions  with  contractors.  The 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA),  as  another  instance,  was  signed 
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into  law  in  1990  and  emphasizes  the  education,  training,  and  certification  of  people  in  the 
acquisition  workforce  (AWF).  Of  course,  the  two  leadership  interventions  are  related:  people 
in  the  AWF  need  to  know  how  to  effect  the  kinds  of  efficiency  and  productivity  improvements 
mandated  via  the  BBPI. 

These  characteristics  of  acquisition  emphasize  the  criticality  of  quality  in  the  AWF 
itself:  With  so  much  at  stake,  and  in  such  a  knowledge-intensive  environment,  a  high-quality 
workforce  is  essential  to  competent  and  professional  acquisition  performance. 

These  characteristics  also  elucidate  the  central  role  played  by  people  and 
organizations  in  the  AWF.  People  must  be  knowledgeable  and  work  effectively — both  in 
terms  of  their  own  professional  acquisition  activities  and  with  many  others  in  acquisition  and 
customer  organizations — in  order  to  accomplish  key  objectives  and  ensure  timely, 
affordable,  and  responsive  delivery  of  products  and  services  to  fighting  and  support  units,  at 
home  and  abroad.  Indeed,  we  understand  well  how  the  efficacy  of  defense  acquisition  is 
inextricably  dependent  on  workforce  quality.  Hence,  leadership  interventions  along  these 
lines  appear  to  be  highly  appropriate  and  on  target. 

Assessing  the  impact  of  interventions  such  as  these  is  a  challenge,  however 
(Assistant  Secretary  of  the  Navy  for  Research,  Development,  and  Acquisition  [ASN(RDA)], 
2011a,  2011b).  It  is  unclear  whether  the  relatively  recent  BBPI,  for  instance,  has  had 
sufficient  time  to  produce  measurable  impact.  Even  after  two  decades  of  the  DAWIA,  as 
another  instance,  efficacy  remains  challenging  to  assess,  for  many  extant  measures  (e.g., 
number  of  Defense  Acquisition  University  graduates,  procurement  lead  times,  program  cost 
growth)  fail  to  account  for  critical  aspects  of  the  AWF  and  important  impacts  on  acquisition 
performance.  Indeed,  it  is  difficult  to  gauge — much  less  predict — the  impact  of  any 
leadership  interventions  along  these  lines  (e.g.,  how  much  better  the  AWF  has  become,  or 
even  if  it  is  improving  over  time).  Hence,  the  impact  of  any  particular  leadership  intervention 
is  left  largely  to  anecdote  and  optimism.  To  help  trim  acquisition  budgets  and  guide 
leadership,  an  improvement  in  assessing  leadership  initiatives  and  interventions  is  needed. 

Because  acquisition  is  a  knowledge-intensive  endeavor  (Snider  &  Nissen,  2003),  the 
knowledge  stocks  of  people  comprising  the  AWF  represent  likely  indicators  of  quality  (e.g., 
education  levels,  training  courses,  years  of  experience,  certification  levels).  However,  such 
indicators  are  relatively  static,  pertaining  to  levels  of  knowledge  that  change  comparatively 
slowly  (Nissen,  2006a).  In  contrast,  acquisition  laws,  rules,  and  regulations  are  revised 
frequently,  and  acquisition  knowledge  can  change  abruptly  and  render  obsolete  even  huge 
stocks  over  time.  Indeed,  this  dynamic  acquisition  environment  requires  members  of  the 
AWF  to  sustain  career-long  learning  and  knowledge  development  just  to  remain  proficient 
as  acquisition  professionals.  Thus,  as  indicators  of  AWF  quality,  static  knowledge  stocks 
appear  to  be  out  of  phase  with  the  highly  dynamic  nature  of  the  acquisition  environment. 

Moreover,  acquisition  organizations  experience  persistent  flux  (Snider  &  Nissen, 
2003).  We  understand  well  that  no  two  acquisition  projects,  programs,  organizations, 
customers,  or  requirements  are  completely  alike.  Hence,  even  well-educated  and  well- 
trained  people,  with  appropriate  certification  levels  and  years  or  decades  of  acquisition 
experience,  must  continually  learn  afresh  and  expand  their  knowledge  further  with  each  new 
assignment.  Likewise,  it  is  clear  that  most  acquisition  organizations  form  and  reform  with 
new  people  (e.g.,  via  personnel  transfer,  turnover,  retirement,  promotion)  continuously  and 
that  end  customer  needs  shift  perennially  (especially  at  the  tactical  edges  of  warfare 
organizations).  Due  to  such  discontinuous  membership  (Ibrahim  &  Nissen,  2007),  even 
these  educated,  trained,  certified,  and  experienced  people  must  learn  repeatedly  to  trust 
and  work  effectively  with  many  others — each  time  someone  new  joins  or  leaves  a  particular 
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acquisition  organization,  and  each  time  a  novel  product,  service,  or  customer  is  involved. 
Thus,  dynamic  knowledge  also  appears  to  be  an  important  AWF  quality  indicator. 

Further,  the  pace  of  change  in  both  information  technologies  and  military  operations 
causes  this  importance  of  dynamic  knowledge  to  apply  in  particular  where  information 
systems  (IS)  are  acquired  to  support  people  at  the  tactical  edges  of  warfare  organizations. 
Not  only  must  acquisition  personnel  be  competent  in  their  professions — including  the 
acquisition  and  maintenance  of  new  acquisition  knowledge  and  skills — and  continually  learn 
afresh  amidst  constant  organizational  flux,  but  they  must  also  keep  pace  with  incessant 
technological  change  and  satisfy  customers’  dynamic  needs.  Even  highly  competent 
professionals  executing  internal  acquisition  processes  perfectly  can  fail  to  satisfy  end 
customers’  materiel  or  service  needs.  This  presents  a  huge  challenge  in  terms  of  assessing 
AWF  quality. 

Building  on  the  development  and  application  of  Knowledge  Flow  Theory  (KFT)  over 
the  past  couple  of  decades  (see  Nissen,  2006b),  we  have  developed  a  state-of-the-art 
approach  that  enables  us  to  analyze,  visualize,  and  measure  dynamic  knowledge  and 
performance.  This  measurement-based  approach  offers  potential  to  overcome  the 
limitations  of  static  measures,  as  previously  summarized,  by  focusing  inwardly  on  the 
dynamics  of  knowledge  important  to  professional  and  effective  acquisition  performance.  The 
main  idea  is  to  measure  the  dynamic  knowledge  and  performance  of  acquisition  processes 
(e.g.,  within  contracting  and  project  management  organizations).  This  would  represent  a 
substantial  step  forward  in  terms  of  acquisition  research. 

Further,  leveraging  complementary  research  in  command  and  control  (C2;  Nissen  & 
Gallup,  2012),  we  see  potential  to  use  this  same  measurement-based  approach  to  also  look 
outwardly  at  the  dynamics  of  knowledge  important  to  professional  and  effective  warfare 
performance.  Although  the  specific  kinds  of  knowledge  required  for  effective  warfare  will 
clearly  differ  from  those  essential  for  proficient  acquisition,  the  approach  is  similar.  The  main 
idea  is  to  measure  the  dynamic  knowledge  and  performance  of  warfare  processes  (e.g.,  at 
the  tactical  edges  of  military  combat  organizations).  This  would  represent  a  substantial  step 
forward  in  terms  of  C2  research. 

Moreover,  we  seek  to  link  these  inward  and  outward  focusing  approaches  to 
conceptualize  an  operational  proxy  for  AWF  quality:  end  customer  performance.  In  addition 
to  measuring  the  dynamic  knowledge  and  performance  of  key  people  and  organizations 
associated  with  IS  acquisition,  for  instance,  we  wish  to  assess  AWF  quality  by  also 
measuring  the  dynamic  knowledge  and  performance  of  primary  beneficiaries  of  such 
systems  acquisition:  end  customers  operating  at  the  tactical  edges  of  warfare  organizations. 
This  proxy  offers  its  best  potential  to  complement,  not  replace,  other  metrics  in 
conceptualization,  development,  and  use  today,  but  it  arguably  concentrates  on  one  of  the 
most  important  AWF  quality  determinants:  how  acquired  systems  affect  operational 
performance.  Two  fundamental  research  questions  follow  accordingly: 

1 .  How  can  dynamic  knowledge  and  performance  metrics  be  applied  to  assess 
acquisition  workforce  quality? 

2.  How  can  dynamic  knowledge  and  performance  metrics  be  extended  to  the 
tactical  edges  of  warfare  organizations? 

Building  heavily  on  the  exploratory  study  reported  by  Nissen  (2012b),  we  summarize 
fast-changing  IS  acquisition  from  the  perspective  of  warfare  at  the  tactical  edge,  and  we 
discuss  dynamic  knowledge  and  performance  measures  to  both  complement  and  contrast 
with  extant,  engineering-oriented  metrics  used  to  specify  and  assess  most  acquired  systems 
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today.  We  begin  with  a  summary  of  KFT  and  measurement  and  then  follow  with  the 
research  method  guiding  the  study.  Results  follow  and  suggest  considerable  promise, 
particularly  where  acquisition  personnel  and  organizations  can  learn  and  track  how 
changing  system  characteristics  correspond  with  operational  performance  at  the  tactical 
edges  of  warfare  organizations  overtime. 

Background 

The  dynamic  nature  of  knowledge  indicates  that  both  stocks  and  flows  are  important 
(Dierickx  &  Cool,  1989).  Knowledge  stocks  have  been  comparatively  straightforward  to 
measure  historically;  metrics  pertaining  to  education  levels,  training  courses,  years  of 
experience,  certifications,  and  like  knowledge-oriented  factors  are  employed  broadly. 
Alternatively,  knowledge  flows  have  been  comparatively  much  more  difficult  to  assess; 
metrics  pertaining  to  dynamic  knowledge — particularly  at  the  group  and  organization 
levels — are  more  elusive.  The  development  and  application  of  KFT  (see  Nissen,  2006b) 
over  the  past  couple  of  decades  has  augmented  the  set  of  tools  and  techniques  available  to 
analyze,  visualize,  and  measure  dynamic  knowledge  and  performance  in  the  organization. 

KFT  is  founded  on  a  set  of  30  principles  that  characterize  dynamic  knowledge.  Such 
principles  are  actionable  and  empirical,  and  they  support  the  diagnosis  of  workflow  and 
knowledge-flow  process  pathologies,  visualization  of  improvement  interventions,  and 
measurement  of  dynamic  knowledge  and  performance  gains  (Nissen,  2006a).  Dynamic 
knowledge  is  delineated  via  five-dimensional  (5D)  vector  space.  Knowledge-flow  vectors 
carry  measurements  and  elucidate  diagnostic  inferences  pertaining  to  the  people, 
processes,  and  organizations  associated  with  knowledge  work.  Figure  1  illustrates  the  idea. 

Briefly,  the  vertical  axis,  “Explicitness,”  characterizes  the  nature  of  knowledge  along 
a  tacit-explicit  continuum.  Tacit  knowledge  implies  understanding  and  know-how/why,  and  it 
is  associated  most  closely  with  the  experiences  of  people  (e.g.,  stemming  from  job 
assignments,  mentoring,  and  teamwork)  and  routines  of  organizations  (including  culture, 
process,  and  ritual).  Explicit  knowledge  implies  awareness  and  know-who/what/where/when, 
and  it  is  associated  most  closely  with  artifacts  (e.g.,  documents,  formulae,  software). 
Generally,  the  more  tacit  the  knowledge,  the  greater  its  appropriability  and  potential  impact 
on  positive  performance  becomes  (Saviotti,  1998).  One  can  measure  knowledge 
explicitness  using  ordinal,  interval,  or  ratio  scales. 


Figure  1.  5D  Knowledge  Flow  Diagram 
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The  horizontal  axis,  “Reach,”  characterizes  how  broadly  knowledge  is  known  and 
shared  in  an  organization.  Here  we  operationalize  reach  in  terms  of  the  number  of  people  in 
an  organization  who  have  access  to  and  can  employ  any  particular  chunk  of  knowledge,  but 
we  could  view  reach  in  terms  of  organizational  levels  instead  (e.g.,  individual,  group, 
organization,  interorganization).  Generally,  the  broader  the  reach  of  knowledge,  the  greater 
its  amplification  and  potential  impact  on  positive  performance  becomes  (Nonaka,  1994). 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

The  axis  “Life  cycle”  characterizes  what  is  being  done  with  a  particular  chunk  of 
knowledge  at  some  specific  point  in  time.  Here  we  include  three  activities:  (1)  some 
individual  in  the  organization  learns  or  creates  new  knowledge;  (2)  he  or  she  shares  existing 
knowledge  with  or  transfers  it  to  other  people  in  the  organization;  and  (3)  one  or  more 
people  in  the  organization  use  or  apply  existing  knowledge  to  accomplish  work.  Generally, 
knowledge  does  not  become  useful  until  it  is  used  or  applied  (Pfeffer  &  Sutton,  1999). 
Measurements  can  be  made  using  categorical  or  ordinal  scales. 

Because  visualization  beyond  three  dimensions  is  difficult,  we  represent  the 
dimension  “Flow  time”  in  terms  of  the  thickness  of  lines  used  to  delineate  vectors.  As  shown 
in  the  key  to  the  right  of  Figure  1,  relatively  thin  lines  are  used  to  delineate  short  and  fast 
knowledge  flows,  whereas  comparatively  thick  lines  represent  knowledge  that  takes  a  long 
time  and  flows  slowly.  Generally,  the  more  quickly  that  knowledge  flows  (e.g.,  across 
people,  organizations,  places,  times),  the  greater  its  potential  impact  on  positive 
performance  becomes  (Nissen,  2002).  Measurements  can  be  made  using  ordinal,  interval, 
or  ratio  scales. 

The  dimension  “Power”  is  represented  similarly  in  terms  of  line  style  used  to 
delineate  knowledge-flow  vectors.  Knowledge  that  flows  with  relatively  low  power — this 
corresponds  with  relatively  low  performance  levels  of  organizational  activities  enabled  by  the 
knowledge — is  delineated  through  orange,  dotted  lines,  whereas  knowledge  flows  exhibiting 
high  power — and  hence  enabling  high  performance — are  delineated  via  purple,  solid  lines. 
Measurements  can  be  made  using  ordinal,  interval,  or  ratio  scales. 

Integrating  these  five  dimensions  graphically  and  analytically  generates  a  5D  vector 
space  to  examine  dynamic  knowledge.  Such  5D  space  and  examination  schemes  are 
completely  general:  they  can  be  applied  to  any  dynamic  knowledge  in  any  organizational 
domain  (e.g.,  acquisition,  C2,  software  engineering). 

As  an  example  of  use  and  application,  consider  Figure  2,  which  illustrates  an 
important  knowledge  flow  desired  by  the  organization.  Point  A  represents  one  individual  in 
the  organization  who  learns  something  new  (to  that  organization)  or  creates  entirely  new 
knowledge.  In  terms  of  the  5D  space,  this  represents  tacit  knowledge  that  is  created  by  an 
individual  (i.e.,  one  person);  hence,  its  position  at  the  bottom-back  corner  of  the  diagram. 
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Figure  2.  Knowledge  Creation  and  Application  Needs 


In  the  acquisition  domain,  for  instance,  consider  that  such  new  knowledge  could 
pertain  to  a  technique  for  reducing  the  acquisition  time  for  an  important  IS  needed  in  the 
field.  Because  information  technology  (IT)  advances  so  quickly — outpacing  the  ability  of 
many  acquisition  organizations  to  develop  and  field  systems  responsively — the  organization 
views  this  new  knowledge  created  at  Point  A  as  important,  and  it  would  like  to  see  such 
knowledge  shared  with  and  applied  by  all  100  people  in  that  organization  who  work  with  IT. 

Such  application  by  100  people  in  the  organization  is  represented  by  Point  B.  The 
thin,  purple,  solid  vector  connecting  Points  A  and  B  represents  the  desired  knowledge  flow: 
the  organization  wishes  for  such  knowledge  to  flow  quickly  and  with  high  power  (e.g., 
enabling  all  100  people  at  Point  B  to  work,  within  one  day,  at  the  same  performance  level  as 
the  innovative  individual  at  Point  A).  This  represents  a  5D  knowledge  flow  vector.  A  question 
mark  in  the  figure  next  to  the  vector  indicates  that  such  a  fast,  powerful  knowledge  flow  is 
desired  by  the  organization,  but  it  is  unclear  which,  if  any,  organizational  process  can  enable 
it. 


This  leads  to  Figure  3,  which  depicts  a  ridge,  or  obstruction,  that  prevents  knowledge 
from  flowing  quickly  and  powerfully  from  Points  A  to  B  as  desired  by  the  organization. 
Practically,  the  organization  lacks  a  process  for  such  quick  and  powerful  knowledge  to  flow 
directly  as  delineated  in  Figure  2.  Indeed,  most  organizations  do  lack  such  a  process 
(Nissen,  2006a).  Some  other  approach  to  sharing  and  applying  the  important  IT  acquisition 
knowledge  is  required. 
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Figure  4  delineates  two  alternate,  archetypical  knowledge  flows  corresponding  to 
processes  that  are  within  this  organization’s  capabilities.  (We  say  archetypical  because  most 
organizations  employ  these  classic  processes  routinely,  and  because  they  present  a  vivid 
contrast  in  terms  of  how  dynamic  knowledge  flows.)  One  knowledge  flow  is  depicted  in 
terms  of  a  relatively  fast  (i.e.,  thin  lines)  but  low-power  (i.e.,  orange,  dotted  lines)  vector 
series;  this  first  flow  is  associated  with  explicit  knowledge  and  utilizes  one  or  more  IS  for 
knowledge  articulation  and  distribution  in  explicit  form.  The  other  is  delineated  via  a 
comparatively  slow  (i.e.,  thick  lines)  but  high-power  (i.e.,  purple,  solid  lines)  vector;  this 
second  flow  is  associated  with  tacit  knowledge  and  utilizes  one  or  more  human-centered 
approaches  to  knowledge  sharing  (e.g.,  group  interaction,  mentoring,  personnel  transfer). 


Figure  4.  Alternate  Archetypical  Knowledge  Flows 

In  some  greater  detail,  the  first  knowledge  flow  consists  of  three  vectors.  The  first 
vector  is  represented  by  a  vertical  line  arising  from  Point  A.  This  vector  depicts  the  individual 
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at  Point  A  articulating  his  or  her  new,  tacit  knowledge  via  an  IS  so  that  it  can  be  shared 
electronically.  Such  articulation  (e.g.,  consider  writing  a  procedure,  developing  a  training 
course,  posting  to  an  intranet  or  social  networking  site)  tends  to  be  somewhat  time- 
consuming,  hence  the  relatively  thick  line.  Articulating  knowledge  in  explicit  form  also  tends 
to  dilute  the  knowledge  in  terms  of  power.  Reading  a  book,  for  instance,  about  how  to 
accomplish  important  acquisition  tasks  (e.g.,  contract  negotiation,  risk  assessment, 
balancing  program  cost  and  schedule  with  performance)  is  not  the  same  as  having  direct 
personal  experience  accomplishing  those  tasks,  hence  the  orange,  dotted  line. 

Once  articulated  in  explicit  form,  however — particularly  via  IS — the  knowledge  can  be 
shared  very  broadly  (e.g.,  organization-wide)  and  very  quickly  (e.g.,  within  seconds),  albeit 
with  diluted  power,  hence  the  thin,  orange,  dotted  line  at  the  top  of  the  diagram.  Indeed,  one 
could  consider  this  broad  and  fast  flow  as  additive  to  the  organization’s  express  acquisition 
body  of  knowledge  (BOK),  which  we  note  at  the  top-right  of  Figure  4.  Such  an  explicit  BOK 
can  then  be  accessed  quickly  and  applied  in  turn  by  all  100  people  in  the  organization.  This 
articulated,  explicit  knowledge  remains  relatively  diluted  and  less  powerful,  nonetheless,  so 
application  at  Point  B  would  not  support  the  same  performance  level  as  at  Point  A,  hence 
the  thin,  orange,  dotted  line  descending  down  to  Point  B. 

Alternatively,  the  second  knowledge  flow  consists  of  a  single  vector,  although  it 
curves  and  bends  through  the  tacit  knowledge  plane  at  the  bottom  of  Figure  4.  This  vector 
depicts  the  individual  at  Point  A  applying  his  or  her  new,  tacit  knowledge  and  then  sharing  it 
with  some  number  of  other  people  (say,  10  people,  as  illustrated  in  Figure  4)  through  one  or 
more  techniques,  such  as  extended  group  interaction,  mentoring,  or  personnel  transfer  to 
work  directly  with  different  coworkers  across  the  organization. 

Once  each  of  these  10  people  has  learned  the  new,  tacit  knowledge,  then  all  of  them 
can  continue  the  process  and  share  it  using  similar  techniques  (e.g.,  group  interaction, 
mentoring,  or  personnel  transfer)  with  others.  Through  such  a  process,  100  people  (i.e. ,  10 
people  each  sharing  with  another  10  people)  can  learn  this  new,  tacit  knowledge  to  the 
extent  necessary  for  powerful  application  at  Point  B.  This  knowledge  flow  is  depicted  by  a 
thick  vector  to  indicate  that  it  occurs  comparatively  slowly,  but  such  vector  is  also  delineated 
by  a  purple,  solid  line  to  show  that  the  corresponding  knowledge  has  high  power  and 
enables  knowledge-based  action  at  the  same  performance  level  as  the  individual  who 
created  it  at  Point  A. 

The  key  is  that  one  can  measure  these  five  dimensions  of  knowledge — whether  via 
explicit  or  tacit  flows — and  relate  them  to  the  corresponding  knowledge-based  process 
performance  by  people  in  the  organization.  Indeed,  by  correlating  such  dynamic  knowledge 
measures  with  performance  metrics,  one  can  develop  a  model  capable  of  analyzing, 
visualizing,  and  even  predicting  process  performance  based  on  knowledge  flow  patterns. 

Of  course,  many  diverse  combinations  of  these  archetypical  knowledge  flows  are 
possible  too,  yet  most  knowledge  flows  are  likely  to  reflect  some  aspects  of  these  two 
dynamic  patterns  (Nissen,  2006b).  Through  empirical  analysis  and  calibration  of  specific 
knowledge  flowing  through  any  particular  organization  in  the  field,  one  can  correlate  5D 
dynamic  knowledge  flows  with  work  performance,  resulting  in  a  model  capable  of 
measurement  and  prediction.  Through  this  technique,  we  are  working  to  assess  AWF  quality 
in  terms  of  dynamic  knowledge  flows. 

Research  Method 

The  first  research  question,  articulated  previously,  includes  a  “how”  interrogative  and 
suggests  that  a  qualitative  method  may  be  most  appropriate  to  investigate  it  (Yin,  1994). 
Despite  the  generality  of  KFT  and  the  5D  space  described  in  the  previous  section,  applying 
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the  corresponding  analytic,  visualization,  and  measurement  techniques  to  assess  AWF 
quality  requires  acquisition  domain  knowledge  in  general  and  process-specific 
understanding  in  particular.  We  need  to  study  one  or  more  specific  acquisition  processes  in 
detail  in  order  to  apply  the  techniques  and  assess  workforce  quality.  The  case  study  method 
is  highly  appropriate  for  an  investigation  along  these  lines  (Benbasat,  Goldstein,  &  Mead, 
1987;  Eisenhardt  &  Graebner,  2007;  Yin,  1994),  and  we  conduct  just  such  a  case  study  in 
parallel  with  the  investigation  reported  here. 

The  second  research  question,  stated  previously,  also  involves  a  “how”  interrogative, 
and  it  likewise  suggests  a  qualitative  method.  However,  this  second  question  calls  for  an 
extension  of  dynamic  knowledge  and  performance  measurement  out  to  the  tactical  edges  of 
warfare  organizations  and  hence  is  much  more  exploratory  from  an  acquisition  perspective. 
Because  we  seek  an  operational  proxy  for  AWF  quality,  we  investigate  dynamic  knowledge 
and  performance  through  explicit  examination  of  three  warfare  organizations  and  processes 
that  are  far  removed  from  core  acquisition. 

One  organization  operates  within  a  U.S.  Navy  fleet  and  has  units  deploying 
rhythmically  to  war  zones  and  other  areas  overseas.  A  second  organization  operates  within 
a  Navy  systems  command  but  concentrates  on  ensuring  the  readiness  of  this  same  fleet. 
The  third  organization  permeates  functionally  throughout  naval  operations  and  is 
responsible  for  information  dominance.  By  interacting  with  knowledgeable  representatives 
from  each  of  these  three  organizations — and  it  is  very  important  to  note  that  these  are 
warriors  and  other  operational  personnel,  not  acquisition  professionals — we  gain 
considerable  insight  into  the  key  knowledge  dynamics  associated  with  warfare  at  the  tactical 
edges. 

Further,  by  triangulating  between  these  three  organizations,  we  identify  a  critical, 
knowledge-intensive  process  that  can  be  represented  with  sufficient  fidelity  and  granularity 
to  suggest  feasible  application  of  our  dynamic  knowledge  and  performance  measures.  The 
process  has  the  somewhat  unwieldy  name  Tasking,  Collection,  Processing,  Exploitation, 
and  Dissemination,  to  which  we  refer  simply  by  its  acronym  TCPED.  In  the  results  that 
follow,  we  delineate  the  TCPED  process  and  seek  to  apply  our  dynamic  knowledge  and 
performance  measures  to  it.  We  then  attempt  to  interpret  such  application  and  to  elucidate 
insight  into  assessing  AWF  quality  via  proxy. 

Results 

Results  from  this  exploratory  investigation  center  on  delineating  the  TCPED  process, 
elaborating  an  insightful  subprocess  in  detail,  and  applying  our  dynamic  knowledge  and 
performance  measures  to  it.  We  discuss  these  in  turn  and  then  focus  on  elucidating  insight 
into  AWF  quality. 

TCPED 

TCPED  does  not  represent  a  new  operational  process  per  se,  but  with  the  U.S. 
Navy’s  relatively  recent  creation  of  its  Information  Dominance  Corps,  it  has  attracted 
considerable  attention  as  a  critical  complement  to  the  find,  fix,  target,  and  track  (F2T2) 
process  associated  broadly  with  combat  operations.  The  key  F2T2  issue  remains 
“knowledge — finding  the  targets”  (Keeter,  2004),  and  as  a  knowledge-intensive  process, 
TCPED  addresses  this  issue  directly,  and  hence  represents  a  promising  target  of  study. 

Given  the  knowledge-intensive  nature  of  TCPED,  its  execution  is  enabled 
fundamentally  by  IT,  and  IS  are  acquired  routinely  with  the  goal  of  enhancing  warfare 
efficacy.  This  nature  provides  an  excellent  link  back  to  our  fundamental  research  question 
and  interest  in  the  AWF.  From  the  operational  perspective  of  TCPED  participants  at  the 
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tactical  edges  of  organizations,  IS  acquired  and  fielded  to  enhance  warfare  efficacy  should 
accomplish  just  that:  enhance  warfare  efficacy.  Further,  such  efficacy  enhancement  should 
be  measurable. 

The  problem  is,  it  is  difficult  to  understand — much  less  measure — how  well  any 
particular  warfare  process  is  working,  which  of  many  different  organizational  arrangements 
are  best  across  diverse  missions,  or  how  well  various  IS  enhance  or  impede  the  process. 
Indeed,  when  seeking  to  acquire  new  IT  and  like  technologies  to  enhance  warfare  efficacy, 
system  implementation  can  make  the  operational  processes  worse  in  the  battle  space,  and 
it  is  increasingly  common  for  different  acquired  systems  to  fail  in  terms  of  interoperating 
(Nissen  &  Gallup,  2012). 

Indeed,  modern  warfare  efficacy  requires  a  combination  of  people  and  technologies 
to  enable  warriors  to  leverage  local  knowledge  and  seize  emergent  opportunities  to  achieve 
commanders’  intent  across  distributed  organizations.  This  requirement  highlights  further  the 
critical  role  played  by  TCPED,  which  seeks  to  enable  commanders  and  warriors  at  the 
tactical  edges  to  put  dynamic  knowledge  into  effective  action,  with  or  without  IS  in 
development  or  in  the  field. 

Additionally,  unlike  many  stable,  mature,  and  well-understood  warfare  processes, 
TCPED  remains  in  a  constant  state  of  analysis,  refinement,  and  development.  Hence,  it 
represents  a  rapidly  moving  target  for  IT  development,  and  engineering-oriented  metrics 
used  to  evaluate  most  IS  fail  to  address  how  dynamic  knowledge  translates  into  effective  (or 
ineffective)  action.  Moreover,  with  current  analytical  models  and  metrics,  it  remains  unclear 
how  to  assess  whether  any  particular  refinement  in  the  warfare  process,  new  IS 
implementation,  or  like  change  will  lead  to  increased  TCPED  efficacy  or  whether 
performance  will  degrade  instead.  This  lack  of  clarity  illuminates  a  capacious  gap  between 
the  efficiency  of  IT  acquisition  and  the  warfare  efficacy  of  IS  employment  at  the  tactical 
edge. 

Given  the  dynamic  nature  of  the  TCPED  process,  as  characterized  previously,  we 
bound  the  scope  of  this  exploratory  project  by  concentrating  on  a  particularly  important  and 
knowledge-centric  subprocess:  exploitation.  Such  bounding  enables  us  to  examine,  within  a 
single  exploratory  study,  the  feasibility  of  our  approach  to  measuring  the  dynamic 
knowledge  and  performance  of  this  operational  process  performed  at  the  tactical  edges  of 
naval  organizations.  Follow-on  researchers  can  then  extend  these  promising  results  via 
subsequent  studies  through  the  process  as  a  whole  and,  in  turn,  to  other  warfare  processes 
seeking  to  benefit  from  IT  acquisition. 

TCPED  Exploitation 

Figure  5  delineates  the  principal  tasks  comprising  TCPED  exploitation.  In  this  figure, 
process  activities  are  depicted  as  rectangular  boxes  connected  to  one  another  via  arrows  to 
delineate  the  process  workflow.  Each  process  activity  is  situated  within  a  horizontal  region 
(referred  to  widely  as  swim  lanes)  that  depicts  the  responsibility  of  a  particular  organizational 
group  to  accomplish  it.  For  several  instances,  the  leftmost  process  activities — “Correlate, 
Fuse  Multi-lnt  Info”  “Operations  Environment  Impact”  “Evaluate  Adversary” 

“Develop  Adversary  COA” — are  shown  connected  together  as  responsibilities  of  the 
“Assessor”  group;  the  “Develop  Adversary  COA”  activity  interrelates  with  “Watch  Analyst 
Coordination,”  the  latter  of  which  is  shown  as  the  responsibility  of  the  Joint  Intelligence 
Operations  Center  (JIOC),  and  which  interrelates  in  turn  with  the  Joint  Operations  Center 
(JOC)  activity  “Watch  Coordination.” 
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Figure  5.  TCPED  Exploitation  Process  Flow 

Other  instances  pertain  to  “Assess  Near-Term  Ops  Impact,”  the  output  of  which 
activity  provides  important  knowledge  and  information  to  Operations  (“J3”);  “Daily  Update 
Information”  and  “Propose  New  Focus  Areas,”  the  output  of  which  activities  provide 
important  knowledge  and  information  to  intelligence  management;  and  “Determine 
Emerging  l&W”  and  “Dissem  l&W,”  both  of  which  activities  are  performed  by  and  are  the 
responsibility  of  the  assessor  as  well.  We  omit  graphical  depiction  or  discussion  of  the  other 
TCPED  exploitation  activities  because  our  intent  is  not  to  be  exhaustive  here,  and  these 
should  suffice  for  our  present  purposes. 

In  particular,  discussions  with  the  knowledgeable  people  interviewed  through  this 
research  indicate  that  the  tasks  labeled  “Evaluate  Adversary,”  “Develop  Adversary  COA,” 
and  “Assess  Near-Term  Ops  Impact”  are  especially  important  and  require  considerable  tacit 
knowledge.  Recall  that  tacit  knowledge,  as  powerful  as  it  is,  tends  to  flow  relatively  slowly 
and  narrowly  through  organizations.  This  makes  it  particularly  challenging  to  support  via  IT, 
and  it  provides  an  excellent  focus  for  our  exploration.  Indeed,  the  people  performing  these 
activities  must  develop  substantial,  tacit  knowledge  pertaining  to  adversaries’  capabilities, 
likely  actions,  and  their  consequences  in  terms  of  friendly  forces  and  operations.  Such  tasks 
also  clearly  require  relevant  and  timely  information,  but  knowledge  of  the  adversary  is  key 
here,  and  the  effectiveness  of  these  tasks  can  contribute  greatly  to — or,  if  ineffective,  impair 
instead — commanders’  decision-making  and  warriors’  actions  on  the  tactical  edge. 

By  focusing  on  how  dynamic  knowledge  flows  through  warfare  process  activities 
such  as  these,  and  especially  by  linking  the  activities  to  knowledge-based  actions  enabled 
at  the  tactical  edge,  we  can  examine  how  well  knowledge  is  flowing  and  supporting  tactical 
action.  Specifically,  by  integrating  the  organizations,  personnel,  and  activities  included  in  the 
exploitation  process  diagrammed  in  Figure  5  with  key  dimensions  from  KFT,  we  seek  to 
identify  critical  paths  in  the  process  where  knowledge  is  flowing  well  and  appropriately,  as 
well  as  identifying  blocked  paths  where  it  is  not,  and  we  strive  to  use  our  dynamic 
knowledge  and  performance  metrics  to  help  overcome  any  disconnects  between  IT 
acquisition  and  warfare  efficacy. 


m  khsiikm 
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Dynamic  Knowledge  Flows 

Through  detailed  analysis,  we  can  delineate  the  principal  knowledge  flows  enabling 
TCPED  exploitation.  Taking  Develop  Adversary  COA  as  an  express  example,  the  people 
performing  this  activity  rely  fundamentally  on  experience-based  tacit  knowledge  (e.g., 
military  tactics,  adversary  capabilities,  organizational  vulnerabilities).  Although  formal 
training  courses,  professional  educational  programs,  and  like  approaches  contribute  to 
these  knowledge  flows,  such  knowledge  is  accumulated  principally  through  direct 
experience  (i.e. ,  on-the-job  training  [OJT]),  often  over  many  years  or  even  decades. 


Figure  6.  Military  Tactics  Knowledge  Flows 


Figure  6  delineates  how  military  tactics  knowledge,  for  instance,  accumulates 
through  cyclic  iteration  between  applying  one’s  existing  tacit  knowledge  (labeled  “Tactics  K 
application”  in  the  figure)  and  learning  from  the  resulting  experience  (labeled  “Tactics  K 
creation”  in  the  figure).  We  locate  this  cyclic  knowledge  flow  vector  at  the  individual  level  of 
reach,  indicating  that  the  Develop  Adversary  COA  activity  is  conducted  in  this  case  by  a 
single  individual.  Were  multiple  people  to  engage  jointly  in  assessments  such  as  this,  we 
would  simply  relocate  the  corresponding  knowledge  flows  to  the  group  level,  with  the  same 
basic  pattern  persisting. 

Consistent  with  our  previous  discussion,  one  can  observe  from  Figure  6  how  the 
vector  for  knowledge  application  is  relatively  thin,  denoting  that  the  flow  is  correspondingly 
fast;  yet  this  vector  is  delineated  via  a  purple,  solid  arrow,  denoting  that  the  flow  reflects 
powerful,  tacit  knowledge.  That  is,  once  tacit  knowledge  has  been  acquired  overtime,  it  can 
be  applied  relatively  quickly.  In  partial  contrast,  the  complementary  vector  for  knowledge 
creation  is  comparably  thick,  denoting  that  the  knowledge  acquisition  flow  is  relatively  slow; 
yet  this  vector  is  also  delineated  via  a  purple,  solid  arrow,  similarly  denoting  that  the  flow 
reflects  powerful,  tacit  knowledge. 

Continuing  with  the  Develop  Adversary  COA  example,  the  people  performing  this 
activity  also  rely  on  a  situated  understanding  of  the  organization’s  current  mission- 
environment  context,  the  adversary  evaluation  synthesized  in  the  preceding  exploitation 
process  step,  and  contemporaneous  knowledge  regarding  both  current  and  future 
operations  being  conducted  and  planned,  respectively,  by  the  organization.  Knowledge 
flowing  to  enable  these  process  activities  follows  somewhat  different  patterns  than  those 
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activities  pertaining  to  military  tactics.  In  particular,  these  latter  knowledge  flows  involve 
interactions  across  different  organizational  groups,  and  they  involve  both  tacit  and  explicit 
knowledge. 

For  instance,  Figure  7  delineates  three  knowledge  flows  associated  with  tacit 
knowledge  sharing  and  intergroup  accumulation.  The  leftmost  cyclic  vector  (labeled 
“Individual  K  accumulation”)  is  comparable  to  that  discussed  previously  in  Figure  6,  except 
that  instead  of  military  tactics  knowledge,  it  pertains  to  the  latter  knowledge  flows  (e.g., 
associated  with  current  mission-environment,  adversary  evaluation,  and  current  and  future 
operations).  We  continue  to  focus  on  individual  knowledge  accumulated  by  a  single 
person — in  this  case,  within  the  assessor  group — but  notice  that  we  include  a  similar  cyclic 
vector  located  at  the  intergroup  level. 


Figure  7.  Tacit  Knowledge  Sharing  and  Intergroup  Accumulation 

This  latter  vector  (labeled  “Intergroup  K  accumulation”)  reflects  tacit  knowledge 
accumulating  across  different  organizational  groups;  multiple  individuals  from  a  variety  of 
groups  work  and  learn  from  their  experiences  together.  The  intergroup  vector  follows  the 
same  cyclic  pattern  as  that  seen  with  individual  OJT,  only  at  a  higher  organizational  level. 

As  with  individual  knowledge  accumulation,  this  intergroup  accumulation  is  delineated  by  a 
cyclic,  purple,  solid  vector  reflecting  knowledge  application  and  creation  occurring  at  two 
different  rates:  quickly  and  slowly,  respectively. 

A  third  vector  (labeled  “Tacit  K  sharing”)  links  the  other  two.  Such  tacit  knowledge 
sharing  reflects  individuals — who  accumulate  knowledge  (especially  via  OJT)  within  their 
separate  groups — sharing  knowledge  with  people  in  other  groups  through  conversation, 
dialogue,  face-to-face  (F2F)  interaction,  and  like  means.  The  two-headed  arrow  included 
with  this  sharing  vector  depicts  knowledge  flowing  bi-directionally:  individuals  share 
knowledge  across  groups  in  the  organization,  and  they  also  learn  through  this  knowledge 
process. 

As  with  the  two  cyclic  vectors  delineated  and  discussed  previously,  knowledge  flows 
corresponding  to  such  tacit  sharing  are  depicted  with  a  purple,  solid  vector  to  designate 
powerful  tacit  knowledge,  and  the  vector  is  depicted  with  a  relatively  thick  line  to  indicate 
that  tacit  knowledge  flows  across  organizational  groups  tend  to  accumulate  relatively  slowly. 
Flowever,  we  depict  this  sharing  vector  with  a  line  that  exhibits  intermediate  thickness;  that 
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is,  the  vector  is  thicker  than  the  application  vector  lines — suggesting  that  tacit  knowledge 
application  flows  across  groups  (e.g.,  in  a  matter  of  days,  weeks,  and  months)  more  slowly 
than  via  individual  application  (e.g.,  in  a  matter  of  minutes,  hours,  and  days) — but  thinner 
than  the  creation  vectors — suggesting  that  such  cross-group  knowledge  can  flow  more 
quickly  than  can  individual  accumulation  of  experience-based  tacit  knowledge  (e.g.,  in  a 
matter  of  months,  years,  and  decades). 


Figure  8.  Explicit  Organizational  Knowledge  Sharing 

As  another  instance,  Figure  8  delineates  alternate  knowledge  flows  associated  with 
explicit  organizational  knowledge  sharing.  The  leftmost  cyclic  vector  (labeled  “Individual  K 
accumulation”)  is  identical  to  that  discussed  previously  in  Figure  7  (e.g.,  cyclic,  purple,  solid, 
powerful,  tacit).  We  continue  to  focus  on  individual  knowledge  accumulated  by  a  single 
person — in  this  case,  within  the  assessor  group — but  notice  that  we  include  a  three-segment 
flow  (labeled  “Explicit  K  sharing”)  to  depict  knowledge  being  shared  organization-wide  in 
explicit  form. 

This  three-segment  flow  begins  with  a  vertical  vector  rising  up  out  of  the  tacit  plane, 
as  an  individual  (i.e.,  in  the  assessor  group)  articulates  his  or  her  tacit  knowledge  into 
explicit  form  (e.g.,  via  textual  reports,  graphical  sketches,  digital  images).  This  articulation 
can  be  a  time-consuming  process;  hence,  the  corresponding  knowledge  flow  vector  is 
depicted  by  a  relatively  thick  line.  In  addition,  we  understand  that  such  articulated,  explicit 
knowledge  does  not  reflect  the  same  power  level  as  the  tacit  knowledge  used  for  its 
creation;  hence,  the  corresponding  knowledge  flow  vector  is  depicted  by  an  orange,  dotted 
line. 


The  second  vector  comprising  this  three  segment  flow  begins  where  the  first  vector 
terminates.  Once  articulated  in  explicit  form,  such  knowledge  can  be  stored,  replicated,  and 
disseminated  quickly  and  broadly  via  one  or  more  IS  (e.g.,  intranet  document  repositories, 
online  sharing  tools,  common  operational  displays).  This  second  vector  in  the  segment  is 
delineated  by  a  thin  line  to  denote  fast  knowledge  flows,  but  the  line  remains  orange  and 
dotted  to  depict  its  diluted  power.  The  third  vector  in  the  segment  is  also  depicted  by  a  thin, 
orange,  dotted  vector,  which  represents  this  same,  diluted,  explicit  knowledge  flowing  via  IS 
quickly  and  broadly  across  the  organization. 
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Dynamic  Knowledge  and  Performance  Measurement 

Through  detailed  analysis,  we  identify  and  operationalize  three  KFT  metrics  that 
appear  to  be  particularly  insightful  for  our  present  purposes:  knowledge  reach  (i.e. ,  how 
many  people  in  the  organization  share  specific  chunks  of  knowledge),  knowledge  flow  time 
(i.e.,  how  long  it  takes  chunks  of  tacit  and  explicit  knowledge  to  flow  from  where  and  when 
they  are  to  where  and  when  they  are  needed),  and  knowledge  power  (i.e.,  the  performance 
level  of  knowledge-enabled  work).  Continuing  with  Develop  Adversary  COA  as  an  express 
example,  we  can  begin  to  quantify  the  key  knowledge  flows  delineated  previously. 


Table  1.  ROOM  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

Individual  K  Accumulation 

1 

Years 

Very  High 

Intergroup  K  Accumulation 

10 

Months 

High 

Tacit  K  Sharing 

10 

Days 

High 

Explicit  K  Sharing 

100 

Hours 

Diluted 

For  instance,  Table  1  summarizes  rough  order  of  magnitude  (ROOM),  3D  estimates 
for  each  of  the  four  knowledge  flows  delineated  and  discussed  previously  with  respect  to  the 
Develop  Adversary  COA  activity  within  TCPED  exploitation.  In  this  table,  we  approximate 
knowledge  flow  measurements  only  to  an  order  of  magnitude,  but  we  begin  to  illustrate  the 
use  and  utility  of  the  approach,  and  we  outline  a  method  for  obtaining  higher  fidelity 
measurements  in  practice. 

In  the  first  column  of  the  table,  we  list  each  of  the  four  knowledge  flows  discussed 
previously;  and  in  the  other  three  columns,  we  summarize  ROOM  estimates  for  knowledge 
reach,  flow  time,  and  power.  Looking  first  at  individual  knowledge  accumulation,  the  reach  is 
listed  as  1;  this  reflects  our  previous  discussion  of  knowledge  being  accumulated  iteratively 
at  the  individual  level,  hence  unitary  reach.  In  the  table,  flow  time  is  listed  in  order  of 
magnitude  as  “years”  for  comparison  with  the  other  knowledge  flows;  this  reflects  our 
discussion  about  how  deep,  experience-based  tacit  knowledge  (e.g.,  pertaining  to  military 
tactics)  can  require  years  or  decades  to  accumulate.  Power  is  listed  likewise  in  order  of 
magnitude  as  “very  high”  for  similar  comparison  with  the  other  knowledge  flows;  this 
estimate  is  somewhat  definitional,  but  it  reflects  that  experience-based  tacit  knowledge  does 
not  suffer  from  power  dilution,  and  it  is  meant  to  reflect  the  considerable  power  of  tacit 
knowledge  accumulated  over  long  periods  of  time  and  through  abundant  experience. 

Looking  next  at  intergroup  knowledge  accumulation,  rough  estimates  for  this 
knowledge  flow  indicate  that  10  people  can  be  reached  by  it;  this  is  an  order  of  magnitude 
larger  than  that  shown  for  individual  knowledge  accumulation,  and  it  reflects  knowledge 
flowing  to  multiple  people  across  organizational  groups.  The  flow  time  estimated  for 
intergroup  knowledge  flows  is  summarized  as  “months,”  which  is  an  order  of  magnitude 
faster  than  that  for  individual  knowledge  accumulation;  this  reflects  the  comparatively  lower 
level  of  deep  knowledge  associated  with  intergroup  knowledge  and  work  flows,  as  people 
across  groups  interact  principally  via  their  present  assignments — which,  in  this  naval 
context,  generally  span  less  than  a  year.  As  discussed  previously,  the  power  level  is  listed 
simply  as  “high”  to  reflect  that  intergroup  tacit  knowledge  (e.g.,  people  learning  to  work  well 
together  across  groups)  does  not  suffer  from  power  dilution,  but  it  also  reflects  that  the 
power  level  is  not  comparable  to  that  associated  with  deep,  experience-based  knowledge 
accumulated  over  years  of  individual  experience  (e.g.,  pertaining  to  military  tactics). 
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Estimates  for  the  third  knowledge  flow  (i.e.,  tacit  knowledge  sharing)  are  similar  in 
terms  of  reach  (10),  but  they  reflect  more  than  another  order  of  magnitude  reduction  in  flow 
time  (i.e.,  “days”);  this  corresponds  to  the  principle  that  knowledge  sharing  can  be 
accomplished  more  quickly  than  the  associated  knowledge  accumulation  (Nissen,  2006b). 
The  (“high”)  power  level  matches  that  for  intergroup  accumulation  mentioned  previously  and 
for  the  same  reasons. 

In  considerable  contrast,  the  flows  associated  with  the  fourth  knowledge  flow  (i.e., 
explicit  knowledge  sharing)  are  quantitatively  very  different.  We  estimate  the  reach  at  100  in 
the  table,  but  the  knowledge  flows  are  constrained  only  by  the  reach  of  the  network 
infrastructure;  hence,  this  figure  could  be  many  orders  of  magnitude  larger  (e.g.,  consider  a 
report,  through  which  everyone  in  a  100,000  person  organization  has  access  to  the  same 
explicit  knowledge).  The  estimate  for  flow  time  is  similar  in  that  we  list  it  as  “hours”  (e.g., 
principally  to  account  for  the  time  required  to  articulate  knowledge  in  explicit  form),  whereas 
once  made  explicit,  such  knowledge  can  be  shared  in  seconds. 

Moreover,  the  power  level  (“diluted”)  for  this  explicit  knowledge  flow  is  qualitatively 
different  from  that  for  its  tacit  counterparts;  this  is  also  somewhat  definitional,  but  it  indicates 
that  most  people  reading  written  documents,  for  example,  will  not  be  expected  to  perform 
knowledge-based  activities  at  the  same  level  as  the  people  writing  those  documents. 

System  Assessment 

The  remaining  measurement  of  knowledge  power  is  linked  directly  to  performance  of 
the  work  activities  enabled  by  such  knowledge.  In  the  case  of  Develop  Adversary  COA,  to 
continue  our  previous  example,  we  could  approach  such  measurement  via  multiple 
operationalizations.  For  several  instances,  we  could  track  how  much  time  is  required  to 
develop  a  set  of  adversary  COAs  sufficiently  well  for  inclusion  in  a  morning  flag  brief  (i.e., 
appropriate  for  presentation  to  a  flag  officer);  using  the  same  flag  brief  criterion,  we  could 
count  how  many  sufficiently  credible  adversary  COAs  are  developed  within  a  set  time  frame 
(e.g.,  one  day,  week,  or  month);  we  could  ask  the  flag  officer  and  staff  in  question  (including 
the  Chief  of  Staff  and  other  directly  reporting  officers)  to  evaluate  the  quality  of  each 
adversary  COA  presented  (based  on  criteria  of  importance  to  them);  or  we  could  pursue  the 
development  of  other,  likewise  understandable  and  relevant  performance  measures.  Any 
such  performance  measure  can  serve  as  a  quantitative  (and  possibly  multidimensional) 
proxy  for  knowledge  power. 

With  one  or  more  such  measures  in  hand,  we  could  then  establish  a  baseline — 
comprised  of  quantitative  measurements  for  reach,  flow  time,  and  knowledge 
power/performance — for  the  organization  as  it  operates  as  usual.  To  evaluate  some 
particular  IS,  we  could  simply  compare  this  baseline  with  measurements  taken  as  the 
organization  uses  the  IS  under  controlled,  or  at  least  comparable,  conditions.  For  instance, 
say  that  we  wish  to  test  a  prototype  IS  designed  to  improve  tacit  knowledge  sharing  through 
introduction  of  social  media  techniques;  we  could  measure  the  knowledge  flows  both  with 
and  without  such  IS  to  assess  its  impacts. 

Specifically,  using  one  or  more  proxy  measures  as  suggested  previously  (e.g.,  time 
required  to  develop  a  set  of  adversary  COAs  for  a  flag  brief,  how  many  adversary  COAs  are 
developed,  flag  officer  quality  evaluation,  others),  we  could  conduct  an  experiment  in  the 
laboratory  or  in  the  “field”  (e.g.,  on  deployed  ships  at  sea)  and  measure  knowledge  and 
performance  directly.  As  an  experiment  to  compare  performance  with  and  without  the 
prototype  IS,  for  instance,  we  would  ideally  like  to  see  the  same  people,  performing  the 
same  tasks,  in  the  same  environments  and  settings,  at  the  same  times  of  day,  seasons  of 
year,  weather  conditions,  sea  states,  and  other  factors  to  isolate  use  of  that  IS  as  the  only 
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difference.  In  other  words,  one  set  of  dynamic  knowledge  measurements  would  be  taken  for 
performance  in  the  baseline  situation;  a  second  set  of  measurements  would  be  taken  for 
performance  with  a  prototype  IS;  and,  ideally  (e.g.,  with  good  experiment  design  and 
techniques),  the  difference  would  represent  solely  the  effect  of  that  IS. 

With  these  measurements  in  hand,  the  difference  in  task  performance  becomes  an 
operational  measure  of  IS  efficacy;  that  is,  if  the  only  difference  between  experiment  cases 
is  whether  the  prototype  IS  used  or  not,  and  task  performance  is  measurably  better  or  worse 
in  one  case  or  the  other,  then  we  have  a  knowledge-based  assessment  of  how  well  such  IS 
improves  (or  worsens)  work  performance  at  the  tactical  edge  of  the  organization  (e.g., 
TCPED  exploitation).  Moreover,  in  addition  to  using  traditional,  engineering-oriented 
performance  measures  (e.g.,  bandwidth,  technical  reliability,  memory),  this  assessment  can 
be  employed  to  evaluate  the  IS  operationally — and  under  controlled  conditions — not  just 
technically.  The  potential  is  huge. 

Further,  given  sufficient  experience  with  conducting  experiments  along  these  lines, 
this  approach  can  even  be  used  to  specify  new  IT  and  other  systems  to  be  acquired;  that  is, 
in  conjunction  with  using  only  engineering  measures  of  IS  performance,  for  instance,  the 
acquisition  organization  can  specify  improvement  in  operational  task  performance  as  a  key 
criterion  forevaluation.  This  way,  acquisition  personnel  can  conduct  efficient  system 
acquisitions,  and  warriors  on  the  tactical  edges  of  organizations  can  use  systems  that 
improve  their  work  performance.  We  bridge  the  gap  between  acquirer  and  warrior,  and 
everybody  wins. 

Illustrative  Example 

In  this  section,  we  include  an  illustrative  example  of  application  to  a  hypothetical  IT 
system  competition.  We  use  only  representative  values  for  illustration  here,  but  the 
approach  and  associated  techniques  can  be  applied  directly  to  system  competitions  in  the 
field.  For  continuity,  we  continue  with  the  Develop  Adversary  COA  task  discussed 
previously,  and  we  build  upon  the  rough  knowledge  flows  and  measurements  reported 
previously. 

Table  2.  Baseline  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

X-Power 

Tacit  K  Sharing 

10 

20  Hours 

95% 

9.5 

Explicit  K  Sharing 

100 

2  Hours 

5% 

5.0 

Table  2  recapitulates  the  most  relevant  measurements  reported  in  Table  1  for  what 
we  term  the  baseline,  representing  the  Develop  Adversary  COA  task  as  it  is  performed 
today  (i.e.,  sans  new  IS);  that  is,  the  baseline  measurements  are  used  for  comparison  with 
this  same  task  performed  with  the  support  of  two  competing  IS  prototypes:  (1)  a  social 
media  application  designed  to  improve  tacit  knowledge  sharing,  versus  (2)  a  document 
collaboration  application  designed  to  improve  explicit  knowledge  sharing. 

Notice  in  Table  2  that  we  limit  our  summary  to  the  pair  of  knowledge  flows 
associated  directly  with  the  alternate  IS:  tacit  knowledge  sharing  (addressed  by  IS-1)  and 
explicit  knowledge  sharing  (addressed  by  IS-2).  Recall,  from  our  discussion  above,  that  the 
knowledge  flow  corresponding  to  tacit  knowledge  sharing  reflects  individuals — who 
accumulate  knowledge  (especially  via  OJT)  within  their  separate  groups — sharing 
knowledge  with  people  in  other  groups  through  conversation,  dialogue,  F2F  interaction,  and 
like  means.  The  central  idea  of  IS-1  is  to  enable  such  knowledge  sharing  remotely;  that  is, 
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the  IS  intends  to  enable  and  promote  tacit  knowledge  sharing  without  the  need  for  (as 
much)  F2F  interaction. 

Recall,  further  from  our  discussion  above,  that  the  knowledge  flow  corresponding  to 
explicit  knowledge  sharing  reflects  organizational  artifacts  (e.g.,  textual  reports,  graphical 
sketches,  digital  images,  and  like  media)  that  are  stored,  replicated,  and  disseminated 
quickly  and  broadly  via  intranet  document  repositories,  online  sharing  applications,  common 
operational  displays,  and  like  tools.  The  central  idea  of  IS-2  is  to  enable  recipients  of 
assessor  reports  (e.g.,  in  the  JIOC  and  JOC  groups)  to  interact  with  assessors  during  report 
development;  that  is,  the  IS  intends  to  enhance  and  accelerate  explicit  knowledge  sharing 
by  providing  recipients  with  access  to  assessor  draft  reports  and  to  enable  communication 
before  finished  reports  are  released  officially. 

Notice  also  that  we  replace  the  ROOM  estimates  from  Table  1  with  quantitative 
values.  For  instance,  the  “days”  flow  time  estimate  from  above  for  the  tacit  knowledge 
sharing  flow  reads  as  “20  hours”  in  Table  2.  Based  on  observation  and  discussion,  roughly 
20  hours  are  required  for  key  tacit  knowledge  to  complete  its  flows.  Further,  the  “high”  power 
estimate  from  above  reads  as  “95%”  here.  As  such,  10  different  people  outside  the  assessor 
group  (e.g.,  in  the  JIOC  and  JOC)  are  able  to  explain  the  details  of  each  adversary  COA 
from  memory  with  95%  accuracy  on  average;  the  other  way  to  look  at  this  is  that  19  of  20 
people  can  explain  the  details  with  100%  accuracy. 

Similarly,  the  “hours”  flow  time  estimate  from  above  for  the  explicit  knowledge 
sharing  flow  reads  as  “2  hours”  here.  This  indicates  that  roughly  two  hours  are  required  for  a 
high-quality  and  credible  adversary  COA  to  be  articulated,  shared  with,  and  understood  by 
recipients.  Further,  the  “diluted”  power  estimate  from  above  reads  as  5%  here.  As  such,  100 
different  people  outside  the  assessor  group  (e.g.,  in  the  JIOC  and  JOC)  are  able  to  explain 
the  details  of  each  adversary  COA  from  memory  with  5%  accuracy  on  average;  the  other 
way  to  look  at  this  is  that  five  of  100  people  can  explain  the  details  with  100%  accuracy. 

Notice,  finally,  that  we  include  a  fifth  column  in  Table  2  (labeled  “X-Power”)  to 
represent  the  induced  dimension  extended  knowledge  power.  Extended  knowledge  power  is 
calculated  as  the  product  of  knowledge  reach  and  power  levels;  it  reflects  the  combined 
distribution  and  efficacy  of  knowledge  flows.  For  instance,  the  extended  knowledge  power 
for  the  tacit  knowledge  sharing  flow  is  shown  in  Table  2  as  9.5  (i.e. ,  reach  of  10  [times] 
power  of  .95  =  x-power  of  9.5),  whereas  the  value  calculated  for  explicit  knowledge  sharing 
flow  is  shown  as  5.0  (i.e.,  reach  of  100  [times]  power  of  .05  =  x-power  of  5.0). 

This  respective  induction  and  quantification  of  the  extended  knowledge  power 
dimension  and  measure  provide  us  with  a  technique  for  comparing  the  efficacy  of  tacit  and 
explicit  knowledge  flows  directly,  despite  the  significant  differences  between  their  dynamic 
characteristics  and  behaviors  (e.g.,  quick,  broad,  diluted  explicit  flows  versus  slow,  narrow, 
powerful  tacit  flows).  Clearly,  higher  values  are  preferred  over  lower  ones,  but  organizations 
face  trade-offs  regarding  whether  to  emphasize  explicit  or  tacit  knowledge  flows. 

Table  3.  Information  Systems  Supported  Knowledge  Flow  Measurement 


Knowledge  Flow 

Reach 

Flow  Time 

Power 

X-Power 

Tacit  K  Sharing  (IS-1) 

20 

20  Hours 

75% 

15.0 

Explicit  K  Sharing  (IS-2) 

20 

2  Hours 

10% 

2.0 

For  further  illustration,  Table  3  summarizes  these  same  knowledge  flow 
measurements — for  the  same  people,  organizations,  tasks,  and  time  frames — after  the 
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prototype  IS  have  been  implemented  and  trained  with.  This  point  is  important;  one  cannot 
expect  a  new  IS  to  be  used  effectively  and  productively  before  its  users  have  been  trained 
adequately.  (It  is  humorous,  nonetheless,  how  often  one  sees  comparisons  made  without 
adequate  training,  particularly  in  field  experiments.) 

In  the  case  of  tacit  knowledge  sharing  supported  by  IS-1 ,  say  that  the  social  media 
application  enables  twice  as  many  people  to  participate  in  the  conversations  (i.e.,  reach 
extends  to  20)  within  the  same  20-hour  time  frame  (e.g.,  by  obviating  the  need  for 
collocation),  but  the  power  level  decreases  to  75%  (e.g.,  due  to  losses  via  mobile  social 
media  applications).  Despite  the  drop  in  power,  the  extended  reach  would  more  than  make 
up  for  the  loss,  because  of  the  extended  power  increase  to  15.0.  Alternatively,  in  the  case  of 
explicit  knowledge  sharing  supported  by  IS-2,  say  that  the  document-sharing  application 
reduces  to  20  the  number  of  people  who  can  participate  effectively  within  the  same  two-hour 
time  frame  (e.g.,  due  to  interference  by  multiple  people  interacting  with  the  same 
documents),  yet  the  power  level  of  those  who  do  participate  increases  to  10%  (e.g., 
stemming  from  increased  textual  interaction  across  organizational  groups).  Despite  the 
increase  in  power,  the  reduced  reach  would  more  than  offset  the  gain  because  of  the 
extended  power  decrease  to  2.0. 


Table  4.  Comparative  Knowledge  Flow  Measurement 


Knowledge  Flow 

Baseline 

IS  Enabled 

Difference 

Difference 

(X-Power) 

(X-Power) 

(X-Power) 

(Percentage) 

Tacit  K  Sharing 

9.5 

15.0 

+  5.5 

+  58% 

Explicit  K  Sharing 

5.0 

2.0 

-3.0 

-  60% 

In  Table  4,  we  summarize  the  comparative  results  via  four  measurements.  First,  the 
Baseline  X-Power  contrast  between  the  tacit  and  explicit  knowledge  sharing  processes 
reflects  our  result  from  Table  2  (i.e.,  9.5  versus  5.0,  respectively).  Second,  the  IS  Enabled  X- 
Power  contrast  between  these  same  processes  reflects  similarly  our  result  from  Table  3 
(i.e.,  15.0  versus  2.0,  respectively).  Third,  the  Difference  X-Power  contrast  measures  the 
effect  of  incorporating  the  two  IS.  For  instance,  using  IS-1  to  support  tacit  knowledge 
sharing  increases  extended  knowledge  power  by  5.5  (i.e.,  15.0  -  9.5  =  +5.5)  for  a  58%  gain. 
In  contrast,  using  IS-2  to  support  explicit  knowledge  sharing  decreases  extended  power  by 
3.0  (i.e.,  2.0  -  5.0  =  -3.0)  for  a  60%  loss. 

Recall  that  the  knowledge  power  measurement  relates  directly  to  organizational 
performance  at  the  tactical  edge,  on  the  Develop  Adversary  COA  task  in  this  illustrative 
case.  In  addition  to  providing  an  objective  and  quantitative  approach  to  assessing  the 
potential  value  (or  harm)  of  an  IS  of  interest,  the  technique  described  in  this  report  also 
suggests  a  way  to  specify  performance  requirements  for  candidate  IS  of  interest. 

Consider,  for  instance,  if — in  addition  to  whatever  engineering  specifications  are 
desirable  or  customary — the  specification  read  along  the  lines  of,  “the  IS  must  demonstrate 
at  least  a  25%  increase  in  X-Power  measured  during  a  fleet  experimentation  exercise.”  This 
specification  would  arguably  place  considerable  contractor  emphasis  on  improving 
knowledge  flow  and  work  performance  of  users  at  the  tactical  edges  of  the  warfare 
organizations  targeted  for  the  acquisition  and  implementation  of  their  IS.  It  would  also 
appear  likely  to  help  bridge  the  gap  between  acquisition  efficiency  and  warfare  efficacy. 

Building  on  these  results,  one  can  now  apply  the  approach  described  in  this  report  to 
any  number  of  IS  acquisitions  and  use  end  customer  performance  as  an  objective  measure 
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of  IS  efficacy.  This  application  will  require  some  venue  for  (at  least  partially  controlled) 
experimentation  (e.g.,  in  the  laboratory,  via  field  experiments,  phased  or  blocked 
implementation),  but  the  potential  benefit  is  huge.  Moreover,  in  addition  to  using  dynamic 
knowledge  and  process  measurement  as  illustrated  here  for  evaluation,  one  can  leverage 
the  same  set  of  measures  to  specify  IS  in  the  conceptualization,  design,  and  development 
phases.  Essentially,  end  customer  performance  becomes  an  objective  design  consideration 
through  this  revolutionary  approach. 

In  terms  of  measuring  AWF  quality,  this  research  establishes  stronger  and  more 
direct  linkages  between  what  acquisition  personnel  know  (especially  focused  internally  on 
acquisition  organizations  and  processes)  and  what  warriors  on  the  tactical  edges  of 
organizations  need  (especially  IS  that  improve  warfare  efficacy),  and  it  provides  a  set  of 
dynamic  knowledge  and  performance  measures  that  can  be  used  to  bridge  the  gap  between 
acquisition  efficiency  and  warfare  efficacy.  This  measurement  step  alone  offers  potential  to 
improve  the  effectiveness  of  those  acquisition  people  and  organizations  that  implement  the 
approach  described  in  this  report;  hence,  one  new  measure  of  AWF  quality  emerges 
directly:  use  of  dynamic  knowledge  and  process  measures  to  assess  end  customer  efficacy. 
In  other  words,  the  working  hypothesis  is  that  those  acquisition  people  and  organizations 
that  use  this  approach  will  be  more  effective  than  those  that  do  not;  hence,  simply  assessing 
the  extent  to  which  this  approach  is  used  may  become  an  important,  complementary 
measure  of  AWF  quality. 

Further,  results  from  this  research  suggest  that  personnel  in  the  AWF  may  benefit 
from  increased  understanding  of  the  end  customers  for  whom  they  acquire  information  and 
other  systems.  The  acquisition  system  as  a  whole  provides  program  offices,  liaisons,  needs 
determination  and  justification  steps,  milestone  and  oversight  authorities,  operational  testing 
and  evaluation,  and  myriad  other  steps  seeking  to  represent  end  customers.  Nonetheless, 
there  may  be  no  substitute  for  acquisition  personnel  who  understand  their  customers  in 
considerable  detail. 

These  results  do  not  suggest  that  procurement  clerks  should  be  outfitted  with 
helmets,  rifles,  and  boots  and  then  sent  to  the  tactical  edges  of  warfare  organizations,  or 
that  warriors  on  such  tactical  edges  should  be  given  procurement  assignments;  rather,  it 
suggests  that  by  examining  the  key  warfare  processes  performed  at  the  tactical  edges — and 
in  particular,  understanding  the  most  important  dynamic  knowledge  and  performance 
characteristics  of  such  processes — even  procurement  clerks  in  offices  half  a  world  away 
may  gain  important  insight  into  their  end  customers — insight  that  may  lead  to  improved 
workforce  quality  and  that  can  be  measured. 
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Abstract 

A  relatively  new  approach  to  government  procurement — strategic  sourcing — offers 
substantially  increased  efficiency  and  effectiveness  to  those  agencies  that  seek  to  implement 
its  tenets.  Sound  market  intelligence  is  the  foundation  of  effective  strategic  sourcing.  The 
government’s  current  approach  to  market  intelligence  is  ad  hoc,  inconsistent,  and  redundant 
because  information  is  rarely  shared  between  buying  activities.  Additionally,  market  research 
is  treated  as  static,  sought  only  to  support  pre-award  acquisition  decisions.  This  article  offers 
a  new  paradigm  for  market  research/market  intelligence  and  demonstrates  ways  in  which 
continuous  market  research/market  intelligence  will  drive  the  government  to  achieve  desired 
strategic  sourcing  outcomes.  This  article  examines  many  facets  of  strategic  sourcing, 
including  goal  setting,  strategic  cost  management,  and  volume  consolidation  strategies.  The 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  301  - 


article  concludes  with  recommendations  for  enhancing  the  collection  of,  dissemination  of,  and 

response  to  market  research/intelligence. 

Introduction 

The  federal  acquisition  system  promulgates  several  objectives  in  acquisition:  to 
procure  goods  and  services  in  terms  of  cost,  quality,  and  timeliness  that  meet  customer 
needs;  to  fulfill  public  policy  objectives  (e.g.,  socio-economic);  to  minimize  administrative 
costs;  to  ensure  integrity,  transparency,  and  fairness  to  the  public;  and  to  attain  the  best 
value.  The  state  of  the  U.S.  economy  and  the  looming  government  budget  constraints 
elevate  the  relative  importance  of  efficiency  as  a  key  acquisition  objective.  A  relatively  new 
approach  to  government  procurement — strategic  sourcing — offers  increased  efficiency  and 
effectiveness  to  those  agencies  seeking  to  implement  its  tenets.  The  GAO  (2012)  contended 
that  billions  of  dollars  can  be  saved  annually  by  strategic  sourcing,  and  criticized 
government  agencies  for  their  lack  of  commitment  to  and  the  subpar  results  produced  by 
strategic  sourcing. 

There  is  a  reasonable  explanation  for  the  lack  of  results.  Strategic  sourcing  is  not  a 
quick,  easy  panacea.  It  requires  experienced  personnel  with  strong  business  acumen,  a 
disciplined  process,  alignment  of  organizational  goals  and  resources,  leadership,  awareness 
of  the  organizations’  needs  and  the  marketplace’s  capabilities,  and  a  culture  that  rewards 
innovation.  Hence,  sound  market  intelligence  is  the  foundation  of  effective  strategic 
sourcing.  Market  intelligence  can  reveal  whether  goals  (e.g.,  cost  savings/avoidance)  are 
attainable.  Agencies’  resources  are  limited;  market  intelligence  can  help  agencies  conduct 
opportunity  assessments  to  discern  which  products  and  services  should  be  strategically 
sourced.  Additionally,  market  intelligence  can  unveil  which  acquisition  strategies  will  achieve 
the  greatest  efficiencies. 

Unfortunately,  this  area  of  great  need  is  also  an  area  of  great  weakness.  The 
government’s  current  approach  to  market  intelligence  is  ad  hoc,  inconsistent,  and  redundant 
because  information  is  rarely  shared  between  buying  activities.  Additionally,  no  existing 
research  or  policy  addresses  how  to  properly  organize  or  resource  the  collection  and  use  of 
market  intelligence.  Furthermore,  specific  skills  for  determining  needed  information,  finding 
it,  analyzing  it,  and  disseminating  it  are  not  systematically  taught  or  developed  in  the 
government’s  acquisition  workforce. 

Therefore,  the  purpose  of  this  article  is  to  explore  ways  in  which  market  intelligence 
can  help  the  government  achieve  its  desired  strategic  sourcing  outcomes.  It  examines  many 
facets  of  strategic  sourcing,  including  goal  setting,  strategic  cost  management,  and  volume 
consolidation  strategies  and  associated  socio-economic  support.  At  conclusion,  it  should  be 
apparent  that  market  intelligence  need  not  be  just  another  checklist  requirement;  rather,  it 
can  be  a  gateway  to  attaining  significant  results. 

The  article  is  organized  as  follows.  First,  historical  and  other  background  information 
surrounding  market  research/intelligence  (MR/MI)  is  reviewed.  Next,  a  brief  review  of  key 
theoretical  underpinnings  is  provided.  A  new  model  of  MR/MI  is  proposed,  which  is  then 
demonstrated  in  three  strategic  sourcing  applications.  Finally,  conclusions  and 
recommendations  are  offered. 

Background 

Market  intelligence  (a.k.a.,  market  research)  has  been  a  statutory  requirement  since 
the  passage  of  the  Competition  in  Contracting  Act  (CICA)  in  1984,  which  required  the  use  of 
market  research  and  procurement  planning  to  promote  the  use  of  competitive  procedures  in 
federal  contracting  (GAO,  1996).  Congress  reemphasized  the  importance  of  market 
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research  in  1990  with  the  National  Defense  Authorization  Act  for  Fiscal  Year  (FY)  1991 
(GAO,  1996).  The  act  encouraged  the  DoD  to  save  money  and  reduce  cycle  time  by 
procuring  commercial  items.  Furthermore,  the  Federal  Acquisition  Streamlining  Act  (FASA) 
posed  additional  requirements  for  market  research  when  enacted  in  1994  (GAO,  1996).  The 
act  required  federal  executive  agencies  to  conduct  market  research  before  developing  new 
specifications  for  a  requirement  and  before  soliciting  proposals  for  a  contract  expected  to 
exceed  the  simplified  acquisition  threshold.  Additionally,  the  FASA  requires  that  contracting 
officers  use  market  research  to  determine  whether  commercial  items  or  non-developmental 
items  could  meet  their  agency’s  needs  if  the  requirement  was  modified  to  some  extent. 

DoDI  5000. 2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition  Programs  and 
Major  Automated  Information  System  Acquisition  Programs  (USD[AT&Lj,  2002),  requires 
that  market  research  and  analysis  be  conducted  to  determine  the  availability  and  suitability 
of  commercial  and  non-developmental  items  prior  to  the  commencement  of  any 
development  effort,  during  the  development  effort,  and  prior  to  the  preparation  of  any 
product  description  (DoD,  1997).  FAR  Part  10  (2011)  prescribes  policies  and  procedures  for 
conducting  market  research  to  arrive  at  the  most  suitable  approach  to  acquiring,  distributing, 
and  supporting  supplies  and  services  (DoD,  1997). 

The  aforementioned  laws  and  regulations  require  the  accomplishment  of  market 
research.  However,  outside  of  a  push  for  commercial  items  and  services,  the  laws  and 
regulations  offer  little  in  terms  of  the  quality  of  market  research  and  how  this  affects 
acquisition  outcomes  (in  both  pre-  and  post-award  contracting  decisions).  The  FAR  (2011) 
offers  little  direction;  Parts  10  and  12  dedicate  a  mere  1 ,477  words  to  the  topic  of  market 
research.  The  DoD  (1997),  Air  Force  Logistics  Management  Agency  (1997),  National 
Aeronautics  and  Space  Administration  (NASA;  1998),  and  Headquarters,  Air  Force  Material 
Command  (HQ  AFMC;  2007)  developed  market  research  guides;  however,  they  are 
outdated  and  do  not  address  market  research  needed  to  support  strategic  sourcing. 

Government  agencies  rarely  budget  for  commercially  available  market  intelligence, 
and  no  existing  policy  addresses  how  to  properly  organize  the  collection  and  use  of  market 
intelligence.  Furthermore,  specific  skills  for  finding,  analyzing,  and  disseminating  information 
are  not  systematically  taught  or  developed  in  the  government’s  acquisition  workforce. 
However,  a  study  of  30  large  firms  showed  that  business  and  market  analysis  is  a  necessary 
skill  of  a  world-class  purchaser  (Giunipero,  2000). 

Concerning  market  intelligence,  there  is  a  difference  between  compliance  and 
effectiveness.  Today,  a  contracting  specialist  can  perform  a  cursory  collection  and 
documentation  of  market  intelligence  and  be  compliant  with  the  FAR  but  at  the  same  time, 
forego  value  due  to  the  omission  of  key  information.  Clearly,  mere  compliance  is  insufficient. 
Given  current  fiscal  constraints,  the  federal  government  is  gradually  elevating  the 
importance  of  efficiency — one  of  several  key  goals  of  the  federal  acquisitions  system  (FAR 
Part  1.102,  2011).  Smart,  informed  decisions  in  pre-  and  post-award  contracting  decisions 
strongly  impact  the  efficiency  of  contracted  outcomes.  Market  intelligence  is  the  key  to 
making  better  decisions  that  provide  more  value  to  the  customer  and  to  the  taxpayer. 

Market  intelligence  also  contributes  to  the  development  of  reliable  cost  estimates  and 
budgets  (Denali  Group,  2009).  The  need  for  market  intelligence  does  not  stop  upon  contract 
award;  it  also  supports  the  negotiation  of  post-award  matters,  such  as  changes  and  dispute 
resolution,  and  is  essential  throughout  the  life  of  the  contract  (Leenders,  Johnson,  Flynn  & 
Fearon,  2006).  Agencies  must  ensure  that  previously  negotiated  prices  remain  fair  and 
reasonable  prior  to  exercising  options.  The  more  critical,  valuable,  complex,  and  risky  the 
procurement,  the  more  important  market  intelligence  becomes  in  order  to  craft  a  contract 
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that  manages  performance  risk,  maximizes  contractor  performance,  balances  financial  risk 
to  both  parties,  and  meets  agency  needs.  Figure  1  lists  contracting  processes  that  require 
valid  and  complete  market  intelligence  in  order  for  acquisition  teams  to  make  optimal 
business  decisions. 


1 .  The  number  and  identity  of  capable 

30. 

Appropriate  supplier  performance  metrics 

suppliers 

2.  The  number  and  identity  of  capable  small 

31. 

Engaging  existing  commercial  logistics  and 

business  suppliers  by  socio-economic 

maintenance  support  infrastructures  to 

category 

decrease  total  life-cycle  support  costs 

3.  Cost  drivers 

4.  The  nature  of  customarily  offered  products 

32. 

Whether  a  reverse  auction  is  appropriate 

and  services 

33. 

Required  buyer  financing 

5.  Current  market  costs  and  prices 

34. 

Market  discounts  or  rebates 

6.  Inflation/deflation  rates 

35. 

Applicable  laws  and  regulations 

7.  Typical  evaluation  criteria  used  to 

36. 

Risks  of  particular  suppliers  based  on  their 

discriminate  between  offers 

record  of  performance 

8.  The  structure  of  the  marketplace 

37. 

Customary  profit  margins 

9.  Analysis  of  the  industry 

38. 

Typical  overhead  rates 

10.  Power  positions  of  the  prospective 

39. 

Existing  government  contracts 

suppliers  relative  to  the  buyer 

40. 

Identify  conflicts  of  interest 

11.  Customary  terms  and  conditions 

41. 

Macro-  and  micro-economic  indicators 

12.  Incentives  that  effectively  motivate  supplier 

42. 

Improve  spend  analysis  by  identifying 

performance 

mergers  and  acquisitions 

13.  Customary  payment  terms 

43. 

Production  rates 

14.  Intellectual  property  rights 

44. 

Assess  supply  and  demand 

15.  Typical  contract  types 

45. 

Labor  rates 

16.  Contract  line  item  structures 

46. 

Inventories 

17.  Contract  durations 

47. 

Data  needed  for  SWOT  analysis 

18.  Customary  surveillance  methods  and 

48. 

Assess  market  share  held  by  prospective 

frequencies 
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Figure  1.  Pre-  and  Post-Award  Demands  for  Market  Intelligence 
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Strategic  sourcing  is  “a  collaborative  and  structured  process  of  analyzing  an 
organization’s  spend  and  using  the  information  to  make  business  decisions  about  acquiring 
commodities  and  services  more  efficiently  and  effectively”  (Office  of  Management  and 
Budget  [OMB],  2005).  In  strategic  sourcing,  requirements  are  aggregated,  contract  values 
are  increased,  customers  per  contract  are  increased,  and  suppliers  are  rationalized.  Hence, 
complexity,  value,  risk,  and  importance  increase  with  strategic  sourcing.  In  order  to  save 
money,  government  acquisition  members  must  focus  more  precisely  on  the  cost  drivers  of 
the  market,  necessitating  more  robust  intelligence. 

Commercial  sector  firms  have  long  recognized  the  importance  of  market  intelligence 
to  effective  supply  management.  Successful  market  intelligence  can  become  a  firm’s 
competitive  advantage  (Porteous,  201 1 ).  Many  firms  staff  business  intelligence  cells  that 
feed  commodity  councils  with  key  information  and  data  (Ashenbaum  &  Pannelle,  2007; 
Zsidisin,  2005).  One  firm  saved  $194  million  through  the  collection  and  use  of  market 
intelligence  (Zsidisin,  2005). 

Literature  Review 

Market  Research/Intelligence 

Market  research  is  the  continuous  process  of  collecting  information  (i.e.,  market 
intelligence)  to  maximize  reliance  on  the  commercial  marketplace  and  to  benefit  from  its 
capabilities,  technologies,  and  competitive  forces  in  meeting  an  agency’s  need  (DoD,  2011). 
Market  research  is  a  vital  means  of  arming  the  acquisition  team  with  the  expertise  needed  to 
conduct  an  effective  acquisition.  Market  research  gathers  current  data  on  existing  market 
sectors  to  identify  potential  sources  of  supply,  commercial  product  characteristics,  market 
characteristics,  commercial  item  standards  and  best  practices,  emerging  technologies, 
vendor  capabilities,  non-developmental  item  solutions,  and  government  leverage 
opportunities  so  that  informed  acquisition  strategy  decisions  can  be  made  (HQ  AFMC, 

2007).  This  market  intelligence  can  be  classified  as  two  types:  strategic  or  tactical. 

•  Strategic  market  intelligence  (a.k.a.,  market  surveillance)  is  an  ongoing 
process,  and  includes  activities  that  the  acquisition  team  performs 
continuously  to  keep  themselves  abreast  of  changes  in  the  marketplace,  such 
as  technological  advances,  process  improvements,  and  available  sources  of 
supply.  The  purpose  of  market  surveillance  is  to  maintain  a  current 
knowledge  base  of  the  depth,  breadth,  and  dynamics  of  the  market  sector 
(HQ  AFMC,  2007). 

•  Tactical  market  intelligence  (a.k.a.,  market  investigation)  is  a  comprehensive 
market  research  survey  conducted  in  response  to  a  specific  acquisition  or 
need.  The  purpose  of  market  investigation  is  to  collect  supporting  data  and 
documentation  to  determine  an  appropriate  acquisition  strategy  (HQ  AFMC, 
2007).  The  appropriate  acquisition  strategy  may  include  pre-  and  post-award 
considerations.  This  may  include  the  following:  planning  for  new  acquisitions, 
deciding  to  exercise  an  option,  and  determining  the  effects  of  key  supplier 
mergers. 

Theoretical  Foundation 

Information  gathering,  dissemination,  and  use  are  grounded  in  market  orientation 
theory  (Kohli  &  Jaworski,  1990).  This  theory  depicts  how  firms  collect  information  regarding 
customer  needs,  disseminate  the  information  within  the  firm,  and  respond  to  the  information 
by  designing  and  offering  products  and  services  that  meet  customer  needs.  A  meta-analysis 
of  market  orientation  (Kirca,  Jayachandran,  &  Bearden,  2005)  shows  that  a  market 
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orientation  increases  innovativeness.  Innovativeness  increases  customer  loyalty  and  quality 
which,  in  turn,  increase  organizational  performance  (profitability).  In  order  to  facilitate 
information  gathering,  dissemination,  and  use,  organizations  need  top-management 
support,  supporting  interdepartmental  dynamics,  and  supporting  organization-wide  systems. 
Departmentalization,  formalization,  and  centralization  hinder  intelligence  generation, 
dissemination,  and  response.  These  are  strong  characteristics  of  government  organizations, 
which  might  hinder  their  effective  use  of  market  intelligence. 

Firms  can  also  benefit  from  collecting  and  using  information  from  suppliers.  “A  supply 
chain  orientation  is  defined  as  the  extent  to  which  there  is  a  predisposition  among  chain 
members  toward  viewing  the  supply  chain  as  an  integrated  entity  and  on  satisfying  chain 
needs  in  an  integrated  way”  (Hult,  Ketchen,  Adams,  &  Mena,  2008,  p.  527).  Such 
information  might  include  supplier  capabilities,  capacities,  constraints,  risks,  strategic  plans, 
and  costs.  Using  the  same  processes  as  market  orientation — information  collection, 
dissemination,  and  response — a  buying  firm  can  improve  its  performance  (customer 
performance,  financial  performance,  internal  process  performance,  and  innovation  and 
learning  performance),  as  was  shown  in  a  study  of  129  firms  by  Hult  et  al.  (2008). 

Essentially,  this  is  what  the  government  does  with  market  intelligence — optimizing  the 
requirement  definition  (i.e. ,  the  need)  by  discovering  what  is  available  in  the  market  instead 
of  defining  needs  based  on  what  was  done  in  the  past.  The  government  has  an  opportunity 
to  improve  performance  by  collecting  the  market  research,  disseminating  it  within  the 
agency,  and  making  appropriate  decisions  by  acting  upon  the  available  information.  All  of 
this  presupposes  that  we  collect  the  right  information  and  make  wise  decisions  from  it.  In 
that  vein,  the  government  can  enhance  credibility  by  using  market  intelligence  to  drive 
acquisition  strategies. 

New  Approaches  to  Market  Intelligence 

A  New  Paradigm 

MR/MI  operates  within  and  through  three  distinct  dimensions:  the  need,  the 
environment,  and  the  plan.  The  need  is  the  definition  of  the  government’s  requirement  and 
is  sought  and  found  in  three  particular  ways:  (1 )  what  we  think  we  need  based  on  previous 
buying  history  or  limited  explanation,  (2)  what  we  actually  need  manifested  as  the  final 
evolved  requirement  through  the  long  government  acquisition  process,  and  (3)  the  optimal 
choice  we  are  unaware  of  or  what  we  could  have  asked  for  if  we  had  understood  our 
environmental  dimension. 

The  environment  is  the  business  and  “battlespace”  in  which  the  government 
operates,  and  is  composed  of  many  factors.  Some  of  these  factors  include  the  industry,  the 
area  of  responsibility,  political  arena,  industry  analysis,  capabilities,  standards,  and  risks. 

The  environment  also  consists  of  socio-economic  issues  and  policies,  as  well  as  external 
considerations  and  risks  (e.g.,  legislation,  national  conflict,  geography,  etc.). 

The  plan  is  the  government’s  strategy  for  how  it  satisfies  its  needs  within  its 
environment,  including,  but  not  limited  to,  the  acquisition  strategy/plan,  source  selection 
plan,  and  small  business  plan.  The  current  model  is  a  serial  process  that  involves  the 
government  doing  the  following: 

•  Step  1:  Determine  the  need  that  is  pushed  by  the  user,  checked  against 
current  supplies  and  previous  purchases,  and  evolved  over  time 
(amendments/changes). 
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•  Step  2:  Assess  the  environment  by  reviewing  vendor  lists,  seeing  where  our 
funds  are  spent,  posting  requests  for  information,  and  consulting  the  Small 
Business  Administration  (SBA). 

•  Step  3:  Develop  the  plan,  such  as  acquisition  plans,  by  holding  acquisition 
strategy  panels,  creating  evaluation  and  incentive  criteria,  determining 
contract  types  and  structures,  coordinating  with  the  SBA,  producing 
government  estimates  and  performance  plans  (e.g.,  quality  assurance 
surveillance  plan),  and  making  option  determinations. 

The  current  model  offers  “too  little,  too  late.”  The  current  approach  takes  a 
reactionary  approach  often  resulting  in  defining  the  need  before  optimizing  the  potential 
solution.  Further,  we  follow  a  serial  approach  in  a  business  environment  that  is  not  linear.  It 
is  global,  multi-dimensional,  and  evolving  faster  than  we  can  react.  We  decide  the  need 
before  we  know  our  environment,  and  the  need  starts  to  change  as  we  develop  our  plan  but 
we  do  not  reassess  the  environment.  When  we  use  immediate  needs  to  drive  MR/MI,  we 
rarely  commit  time  to  reassess.  Finally,  the  current  model  does  not  meet  the  intent  of  FAR 
(2011)  Subpart  10.001(a)  to  conduct  market  research  on  an  on-going  basis.  Current  practice 
is  to  conduct  market  research  as  an  initial  step  to  acquisition  planning  that  is  done  at  the 
beginning  and  not  monitored  after  the  fact. 

The  proposed  model  (see  Figure  2)  recognizes  three  distinct  dimensions  to  be 
assessed  simultaneously  and  continuously,  while  maintaining  a  high  level  of  education  and 
training.  The  need  dimension  involves  having  early  talks  with  management,  leadership, 
approving  offices,  technical  SMEs — as  with  an  early  strategy  and  issues  session  (ESIS) — 
and  functional  users  12  to  24  months  prior  to  an  anticipated  award.  Further,  the  need 
dimension  involves  maintaining  a  robust  spend  analysis  of  current  contract  portfolios  with 
informed  projections  for  future  portfolios,  using  tools  such  as  a  purchasing  portfolio  model  to 
segment  spend  by  type  (Kraljic,  1983).  It  further  involves  understanding  agency  tendencies 
and  constraints  using  tools  such  as  a  strengths,  weaknesses,  opportunities,  and  threats 
(SWOT)  analysis. 

The  environment  dimension  involves  holding  industry  days  and  issuing  requests  for 
information  (RFI)  periodically  to  monitor  new  entrants,  market  trends,  bundling/consolidation 
issues,  and  possibilities.  Other  considerations  include  Porter’s  (1979)  five  forces  analysis,  a 
power-matrix  analysis  (Cox,  2001),  and  a  risk  analysis  (cost,  technology,  performance),  and 
capturing  market  cost  drivers  while  assessing  regulation,  standards,  and  commercial 
practice.  Finally,  the  environment  dimension  must  consider  monitoring  external  issues  such 
as  national  political  trends  and  legal  and  regulatory  developments. 

The  proposed  model  introduces  the  concept  of  an  education  and  training  (E&T) 
cycle,  the  idea  being  that  all  market  intelligence  collected  during  the  continual  processes 
overtime  is  shaped  by  previous  and  current  education  and  training,  and  must  shape  future 
MR/MI  efforts  and  improve  education  and  training. 
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Figure  2.  Proposed  Model  of  Perpetual  Market  Research 


Under  the  proposed  model,  the  MR/MI  process  is  a  synergistic  process  that 
combines  all  dimensions,  and  assesses  how  to  optimize  needs  in  a  changing  environment. 
This  proposed  model  directs  our  focus  to  the  changing  environment  and  being  proactive 
instead  of  focusing  on  reactive,  short-term  needs.  Acknowledging  an  increasingly-rapid  pace 
of  changes  to  our  environment,  and  recognizing  the  evolving  primacy  of  efficiency  as  a 
critical  acquisition  outcome,  the  value  of  this  proposed  model  of  MR/MI  becomes  apparent. 
This  value  can  be  demonstrated  in  many  steps  and  activities  of  the  strategic  sourcing 
process.  The  following  sections  elaborate  on  three  activities  in  which  MR/MI  offers 
opportunities  for  improved  acquisition  outcomes:  goal  setting  and  opportunity  assessment, 
strategic  cost  management,  and  consolidation/bundling. 

Goal  Setting  and  Opportunity  Assessment 

Purchasing  (i.e.,  supply  management)  is  a  strategic  activity  (Carter  &  Narasimhan, 
1996)  due  to  its  ability  to  contribute  to  a  firm’s  competitive  advantage  (Ellram  &  Carr,  1994). 
Two  of  the  most  important  and  implemented  aspects  of  strategic  supply  management  are 
strategic  planning  and  performance  measurement  (Monczka  &  Petersen,  2008).  Firms  that 
develop  supply  management  strategic  plans  typically  set  three-to-five  year  outlooks  with 
goals  linked  to  key  performance  indicators.  Progress  toward  goals  is  measured  as  often  as 
twice  per  month.  It  is  often  said  that  what  gets  measured,  gets  done,  and  that  metrics  drive 
behavior.  Supply  management  leaders  are  responsible  for  setting  and  achieving  appropriate 
sourcing  goals,  and  such  goals  should  feed  into  the  organization’s  overall  goals  and 
strategies.  These  goals  and  metrics  focus  commodity  councils  on  what  is  important.  Goals 
should  be  specific,  measurable,  attainable,  relevant,  and  timed  (Rudzki,  Smock,  Katzorke,  & 
Stewart,  2006). 

But  how  does  a  commodity  director  know  whether  his  or  her  savings,  efficiency,  and 
effectiveness  goals  are  attainable?  Market  intelligence  plays  a  pinnacle  role.  First,  an 
organization  should  benchmark  its  performance  against  its  competitors  and  against  best-in- 
class  organizations  (Rudzki  et  al.,  2006).  Reports,  data,  and  benchmarks  are  often  available 
from  sources,  such  as  AT  Kearney,  McKinsey,  Aberdeen  Group,  CAPS  Research,  Sourcing 
Interest  Group,  Gartner  Group,  IBISWorld,  Forrester,  Market  Reports  Online, 
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MarketResearch.com,  Research  and  Markets,  consultants,  and  various  industry-specific 
trade  associations.  Participating  in  electronic  reverse  auctions  (eRA)  and  buying  consortia 
also  unveil  current  pricing  information.  Second,  routine  comparisons  to  historical  prices  paid 
should  be  made.  If  the  procuring  contracting  officer  were  asked  the  current  prices  paid, 
would  he  or  she  be  able  to  respond  without  opening  the  contract  file?  Third,  prices  could  be 
compared  to  the  producer  price  index  available  from  the  U.S.  Bureau  of  Labor  Statistics. 
However,  note  that  this  index  is  not  always  sufficiently  precise  (Rudzki  et  al. ,  2006). 

Rudzki  et  al.  (2006)  offered  general  ranges  of  savings  by  type  of  spend  (see  Table 
1).  These  benchmarks  can  be  used  not  only  to  set  goals  for  a  commodity  council  but  for 
specific  requirements  as  well.  Note  that  these  levels  of  savings  are  not  unique  to  the  for- 
profit  sector.  Government  buyers  have  achieved  nearly  double  the  savings  (28%)  compared 
to  their  for-profit  sector  counterparts  on  sourcing  improvement  projects  (Husted  &  Reinecke, 
2009).  There  appears  to  be  ample  opportunity  for  the  government  to  improve. 

Often,  organizations  will  have  more  requirements  to  source  than  resources  available 
to  source  them  strategically.  In  this  case,  strategic  sourcing  organizations  must  prioritize 
sourcing  events  (i.e.,  requirements).  One  tool  to  facilitate  these  decisions  is  an  opportunity 
assessment.  Here,  each  requirement  is  assessed  in  terms  of  the  degree  of  difficulty  of 
implementation  and  savings  and/or  performance  impact  (Monczka  &  Petersen,  2008). 
Obviously,  those  requirements  that  are  easier  to  implement  yet  yield  substantial  savings  will 
be  sourced  first.  The  important  point  here  is  that  the  savings  potential  cannot  be  validly 
estimated  without  solid  market  intelligence  that  unveils  opportunities  to  alter  strategies.  This 
requires  near-constant  market  surveillance  and  deep  category  expertise.  High  turnover  will 
cripple  the  ability  to  collect,  disseminate,  and  act  upon  market  intelligence,  that  is,  to  know 
the  market. 


Table  1.  Savings  Opportunity  by  Type  of  Spend 


Type  of  Spend 

Potential  Savings  (%  of 
total  spend) 

Raw  materials 

2-5 

Packaging 

10-20 

Indirect  materials  and  services 

10-20 

Information  technology 

15-30 

Professional  services 

8-15 

Logistics 

7-15 

Media/marketing/promotional  items 

10-20 

Other  indirects 

5-15 

Capital  projects 

7-15 

Strategic  Cost  Management 

An  important  tenet  of  strategic  sourcing  is  strategic  cost  management  (Monczka  & 
Petersen,  2008,  p.  43),  defined  as  “the  identification  and  proactive  management  of  all  costs 
and  associated  cost  drivers  throughout  the  product/service  supply  chain.”  It  “requires 
development,  prioritization  and  implementation  of  strategies  and  processes  to  control, 
reduce  or  eliminate  costs  during  each  phase  of  the  life  cycle”  (Monczka  &  Petersen,  2008,  p. 
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43).  Strategic  cost  management  offers  substantial  opportunities  for  cost  savings  and  cost 
avoidance,  as  illuminated  in  the  following  three  examples.  As  evidenced  in  these  examples, 
market  intelligence  is  essential  to  identify,  quantify,  and  understand  cost  drivers. 

The  first  example  concerns  elevator  maintenance  services.  The  Air  Force’s 
Enterprise  Sourcing  Group  conducted  extensive  market  research  in  2011  (HQ  AFMC,  201 1 ). 
Figure  3  shows  elevator  maintenance  cost  drivers  provided  by  IBIS  World,  a  leader  in 
syndicated  market  research.  Labor  and  profit  account  for  the  majority  of  costs  (Ripley, 

2011).  Employee  compensation  declined  while  industry  profitability  peaked  at  29%  in  2008 
(Ripley,  2011).  Compared  to  similar  industries,  there  may  be  opportunity  to  negotiate  a 
lower  margin.  A  comparison  of  historical  rates  to  prevailing  market  rates  revealed  that  the 
Air  Force  was  paying  18-20%  more  than  other  federal  and  commercial  clients  (HQ  AFMC, 
2011). 
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Figure  3.  Industry  Cost  Breakout 
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Prices  depend  on  cost  drivers,  such  as  the  number  of  units,  type  of  equipment,  age 
of  equipment,  manufacturer,  equipment  usage,  desired  service  call  frequency,  and  location 
of  equipment.  Prices  differ  significantly  by  equipment  types.  Because  traction  elevators 
contain  more  moving  parts  and  maintenance  requirements  than  do  hydraulic  elevators,  their 
cost  is  two  to  three  times  higher.  Additionally,  equipment  age  is  highly  correlated  with  the 
degree  of  required  maintenance  and  repair.  The  Air  Force’s  oldest  elevator  equipment  is  60 
years  old,  with  an  average  age  of  approximately  20  years.  The  equipment  manufacturers 
also  drive  costs.  A  contractor  may  charge  more  to  service  a  wide  variety  of  equipment. 
Contractors  seek  to  offset  the  risks  of  obsolescence  costs  from  servicing  equipment  from 
manufacturers  that  are  no  longer  in  business.  The  frequency  of  service  calls  affects  pricing 
as  well.  Customers  requiring  more  frequent  service  incur  greater  cost  due  the  need  for  on¬ 
site  technician  time  and  associated  travel  expenses.  A  growing  trend  in  the  industry  is 
usage-based  service  rather  than  regularly  scheduled  maintenance.  Relatively  low  usage  by 
the  Air  Force  could  yield  cost  savings  by  converting  to  demand-based  versus  time-based 
service  (HQ  AFMC,  2011). 

As  a  second  example,  consider  a  Fortune  500  firm  that  outsourced  the  supply  and 
management  of  its  service  vehicle  fleet.  The  total  cost  of  ownership  breakdown  (see  Figure 
4)  reveals  the  major  cost  categories  to  be  the  lease  expense,  fuel,  and  vehicle  maintenance. 
However,  the  underlying  cost  drivers  were  the  quantities,  ages,  types  of  vehicles, 
depreciation  rates,  cost  of  capital,  miles  driven,  cost  per  barrel  of  oil,  vehicle  condition,  and 
maintenance  labor  and  parts.  Most  government  acquisition  professionals  would  look  to 
minimize  the  major  cost  categories  but  often  overlook  a  deeper  investigation  of  underlying 
cost  drivers.  For  example,  a  contracting  officer  might  leverage  competition  to  reduce  the 
cost  of  capital,  or,  again  via  competition,  might  influence  offerors  to  seek  the  most  cost- 
effective  national  maintenance  network.  However,  they  may  overlook  other  opportunities  for 
cost  avoidance  via  tenets  of  strategic  sourcing,  such  as  demand  management  and  e- 
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sourcing.  For  example,  prior  to  the  vehicle  strategic  sourcing  event,  the  internal  customer 
defined  the  fleet  needs  as  in  the  past:  The  firm  needed  2,600  service  vans.  By  staying 
abreast  of  developments  in  the  market,  an  astute  commodity  manager  discovered  that  an 
auto  manufacturer  altered  its  strategy  to  sell  one  of  its  models.  This  model  did  not  sell  well  in 
the  consumer  market  (Kiley,  2005);  thus,  the  manufacturer  repositioned  it  as  a  fleet 
vehicle — at  a  steep  discount  compared  to  traditional  vans.  By  collecting  this  market 
intelligence,  disseminating  it  within  the  user  community,  and  acting  upon  it  (i.e.,  switching 
vehicle  types),  the  commodity  manager  saved  approximately  $1  million  on  its  $23  million 
fleet  spend.  Hypothetically,  using  another  savings  lever,  the  commodity  manager  could 
require  the  fleet  management  contractor  to  source  its  fleet  vehicles  from  manufacturers 
using  electronic  reverse  auctions  (eRA),  an  e-commerce  tool  that  typically  saves  buyers 
20%  (Cohn,  Brady,  &  Welch,  2000)  via  online,  real-time  competition  (Hawkins,  Randall,  & 
Wittmann,  2009). 
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Figure  4.  Vehicle  Fleet  Total  Cost  of  Ownership  Analysis 


As  a  final  example,  consider  the  Air  Force’s  attempt  to  strategically  source  security 
guard  services  at  29  installations  in  2004  (Bowman,  Reed,  Hudgens,  &  Searle,  2006).  The 
major  cost  category  was  labor.  The  savings  lever  sought  was  economies  of  scale  by 
consolidating  separate  contracts  at  several  installations.  However,  rigidity  of  the  major  cost 
driver  was  overlooked.  The  labor  rates  were  subject  to  the  Service  Contract  Act  of  1965; 
thus,  the  Department  of  Labor  established  minimum  wage  rates  via  wage  determinations 
(based  on  average  wages  in  each  locale).  These  wage  rates  remained  constant  regardless 
of  the  number  of  employees  hired  under  a  single  contract.  Thus,  while  transaction  costs 
were  somewhat  reduced,  the  resultant  contract  failed  to  yield  meaningful  savings  from 
economies  of  scale.  These  three  examples  highlight  the  importance  of  market  intelligence  in 
strategic  cost  management. 
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Consolidation  Strategies  and  Socio-Economic  Support 

MR/MI  proves  critical  to  bundling  and  consolidation  procurement  strategies.  Both 
bundling  and  consolidation  aggregate  requirements  to  (1)  achieve  volume  savings  from  the 
marketplace,  (2)  reduce  transaction  costs  associated  with  multiple  source  selections  and 
multiple  contracts,  and  (3)  reduce  performance  risks  associated  with  managing  a  greater 
variance  of  performance  across  more  suppliers.  FAR  (2011)  Subpart  2.101  defines  bundling 
as  consolidating  two  or  more  requirements  for  supplies  or  services,  previously  provided  or 
performed  under  separate  smaller  contracts,  into  a  solicitation  for  a  single  contract  that  is 
likely  to  be  unsuitable  for  award  to  a  small  business  concern. 

DFARS  (201 1 )  Subpart  207.170-2  defines  consolidation  of  requirements  as 

the  use  of  a  solicitation  to  obtain  offers  for  a  single  contract  or  a  multiple 
award  contract  to  satisfy  two  or  more  requirements  of  a  department,  agency, 
or  activity  for  supplies  or  services  that  previously  have  been  provided  to,  or 
performed  for,  that  department,  agency,  or  activity  under  two  or  more 
separate  contracts. 

Consolidation  or  bundling  of  requirements  increases  the  scope  of  work  performed  by  the 
contractor.  Because  a  firm’s  revenue  or  number  of  employees  determines  its  small  business 
designation  within  its  industry,  the  increased  scope  can  make  it  more  difficult  to  obtain 
competitive  offers  from  two  or  more  small  businesses.  Subsequently,  consolidated  or 
bundled  procurement  solicitations  may  go  out  as  unrestricted,  requiring  small  businesses  to 
compete  directly  with  large  businesses. 

FAR  (2011)  Subpart  7.107  specifically  addresses  bundling  contract  actions  as  they 
relate  to  small  business.  In  order  to  bundle  requirements,  the  government  must  ensure  that 
it  considers  the  impact  on  small  business  participation  and  the  measurable  benefits  of 
bundling  (e.g.,  quality  improvements,  administrative  or  direct  cost  savings,  etc.).  Additionally, 
FAR  (2011)  Subpart  7.107(a)  states  that  “because  of  the  potential  impact  on  small  business 
participation,  the  head  of  the  agency  must  conduct  market  research  to  determine  whether 
bundling  is  necessary  and  justified.”  The  FAR  establishes  minimum  percentage  savings 
thresholds  for  bundling  to  balance  the  government’s  cost  efficiency  goals  with  socio¬ 
economic  goals.  According  to  FAR  (201 1 )  Subpart  7.1 07(b),  the  agency  may  justify  bundling 
as  compared  to  the  benefits  that  it  would  derive  from  contracting  to  meet  those  requirements 
separately  if  it  results  in  savings  equal  to  or  greater  than 

(1)  ten  percent  of  the  estimated  contract  or  order  value  (including  option)  if 
the  value  is  $94  million  or  less;  or  (2)  five  percent  of  the  estimated  contract  or 
order  value  (including  options)  or  $9.4  million,  whichever  is  greater,  if  the 
value  exceeds  $94  million. 

Due  to  the  perceived  negative  impact  to  small  business,  bundling  and  consolidation 
are  politically  sensitive,  to  say  the  least.  Any  savings  estimates  will  likely  be  scrutinized. 
MR/MI  provides  the  key  information  required  to  quantify  and  substantiate  the  realistic 
savings  potential.  Although  a  solid  business  case  may  justify  bundling  or  consolidation,  such 
a  strategy  may  be  perceived  as  politically  untenable.  Nevertheless,  compelling  savings  and 
performance  improvement  opportunities  may  open  avenues  to  compromises  (e.g., 
consolidated  or  bundled  small  business  set  asides,  partial  small  business  set-asides,  or 
requirement  offsets)  that  offer  a  win-win  outcome. 

The  FAR  and  DFARS  are  very  specific  in  their  requirements  for  bundling  contracts  to 
minimize  the  impact  on  small  businesses.  However,  while  the  information  required  is  clear, 
the  methods  of  collection  are  ambiguous.  Examining  current  and  past  contracts,  contracts  of 
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other  agencies,  industry  best  practices,  academic  articles;  attending  conferences;  or 
conferring  with  third  party  consultants  are  all  valid  methods  of  data  collection.  The  amount  of 
evidence  necessary  to  substantiate  cost  savings  will  rely  on  the  amount  required  by  the 
Head  of  Contracting  Activity.  Additional  considerations  may  exist  within  the  industry  or 
market  further  limiting  bundling.  All  these  issues  must  be  considered  when  performing 
market  research  to  bundle  or  consolidate  contracts.  MR/MI  is  pivotal  in  determining  whether 
a  small  business  can  provide  the  desired  product  or  service. 

An  example  was  the  Air  Force’s  Furnishings  Commodity  Council  (AFFCC)  in  2009. 
The  AFFCC  utilized  MR/MI  to  identify  industry  best  practices,  benchmarked  those  best 
practices,  and  created  business  cases  for  cost  savings  initiatives.  To  identify  the  savings 
opportunity  for  each  business  case,  the  AFFCC  used  a  percentage-of-savings  methodology 
based  on  government  and  commercial  savings  benchmarks,  historical  Air  Force  spend 
analysis  from  FY2000  to  FY2007,  and  furnishings  market  forecast  information. 

The  AFFCC  relied  heavily  on  a  spend  analysis  to  determine  historical  spend  data  on 
which  to  base  the  savings  estimates.  Based  on  the  historical  spend,  the  AFFCC  was  able  to 
forecast  spend  data  from  2009  to  2013.  The  results  of  the  spend  analysis  showed  that  over 
76%  of  furniture  purchases  were  made  from  small  businesses.  Additionally,  market  research 
showed  that  over  50%  of  an  office  furniture  manufacturer’s  cost  structure  was  variable,  and 
that  labor  made  up  the  majority  of  fixed  costs.  This  led  the  AFFCC  to  the  volume  purchasing 
sourcing  strategy.  The  market  research  showed  that  manufacturers  are  attracted  to  volume 
purchases  due  to  the  ability  to  lower  cost  by  fully  utilizing  labor,  which  is  the  second  largest 
component  of  furniture  cost.  As  a  result,  the  AFFCC  utilized  industry  benchmarks  from 
government  and  commercial  sources  to  estimate  five-year  savings  within  three  categories: 
conservative  (3%),  moderate  (6%),  and  aggressive  (9%;  Air  Mobility  Command  [AMC], 
2009). 

The  three  savings  estimate  categories  were  applied  to  three  business  cases  to  show 
cost  savings.  The  business  cases  included  the  following  savings  levers:  develop  Air  Force 
furnishing  standards  and  supporting  policy  (standardization);  develop  centralized  contract 
vehicles  (leverage  volume  to  drive  price  reductions  and  improve  purchasing  efficiency);  and 
acquire  comprehensive  furniture  management  services  consisting  of  seven  categories  to 
include  project  management,  asset  management,  reconfiguration/relocation  management, 
space  planning  and  design,  packaged  furnishings,  asset  maintenance,  and  site  preparation 
and  reconfiguration  (AMC,  2009).  The  market  research  enabled  the  AFFCC  to  conclude  that 
over  a  five-year  period,  furniture  standardization,  a  centralized  contract  vehicle,  and 
comprehensive  furniture  management  services  savings  combine  for  an  estimated  cost 
savings  between  10.6  to  215  or  $41.2  million  to  $81.8  million,  respectively  (AMC,  2009).  The 
conservative  estimates  of  savings  exceeded  the  thresholds  necessary  for  bundling  and 
consolidation. 

The  commodity  team’s  goal  was  to  reduce  life-cycle  costs,  eliminate  duplicate 
efforts,  standardize  requirements,  and  lower  total  ownership  costs.  The  AFFCC  created  a 
standardized  requirements  list  for  all  bases.  This  list  included  basic  specifications  for 
different  types  of  office  chairs  such  as  executive,  executive  guest,  and  side/general  seating. 
Each  requirement  also  had  a  minimum  warranty  that  vendors  would  have  to  guarantee.  The 
idea  was  to  make  the  requirements  as  basic  as  possible  and  to  allow  suppliers  to  quote 
various  options.  Once  they  identified  what  the  requirements  would  be,  the  AFFCC  began  to 
research  the  available  furniture  vendors  in  the  market. 

Most  of  the  furniture  manufacturers,  large  and  small,  used  furniture  dealers  to  market 
and  sell  their  products.  Most  of  these  dealers  are  small  businesses  located  throughout  the 
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country.  Manufacturers  typically  do  not  have  their  own  showrooms.  Some  dealers  only 
specialize  in  certain  manufacturers’  brands,  but  for  the  most  part,  dealers  represent  all 
manufacturers.  One  of  the  methods  used  to  gain  vendor  awareness  was  the  National 
Exposition  of  Contract  Furnishings  (NEOCONs)  world’s  trade  fair  in  Chicago.  Participants  of 
the  trade  show  learn  about  the  latest  designs,  trends  in  fashion,  and  scientific  breakthroughs 
in  chair  ergonomics. 

Through  further  research  and  the  help  of  consulting  firms,  the  Air  Force  determined 
that  63%  of  furniture  manufacturing  was  done  by  the  “Big  Five”  companies.  An  RFI  was 
posted  in  2007,  and  41  responses  were  received.  Most  of  the  distributors  proposed  teaming 
agreements  with  large  manufacturers.  In  2008,  members  of  the  AFFCC  attended  the  2008 
NEOCON.  The  teams  also  learned  what  each  manufacturer’s  production  capacity  was  and 
whether  they  could  handle  the  increased  capacity  of  supplying  the  Air  Force. 

After  thorough  market  analysis  and  research,  the  AFFCC  determined  that  the 
commercial  marketplace  could  fulfill  the  Air  Force’s  needs,  and  that  the  seating  products 
offered  via  the  GSA  schedule  met  the  minimum  requirements.  Through  spend  analysis,  the 
Air  Force  Small  Business  Solution  Center  (AFSBSC)  identified  that  only  23%  of  the 
suppliers  of  office  furniture  were  small  business  non-GSA  manufacturers  (AFSBSC,  2009b). 
However,  the  AFSBSC  found  that  wood  seating  comprised  of  mostly  niche  small  business 
manufacturers  (AFSBSC,  2009b).  In  addition,  the  Air  Force  bought  80%  of  dorm  furnishings 
from  small  businesses  (AFSBSC,  2009a).  Thus,  it  was  determined  that  even  with 
consolidation,  the  AFFCC  would  receive  adequate  small  business  competition  for  Spiral  1 
(wood  seating)  and  Spiral  1A  (dorm  furnishings).  Extensive  MR/MI  gave  the  AFFCC  current 
market  condition  information  necessary  to  make  an  informed  and  substantiated  small 
business  participation  determination  for  some  wood  seating  and  dorm  furnishings  while 
supporting  consolidation  for  office  furniture. 

Conclusion 

The  importance  of  thorough  MR/MI  cannot  be  overstated.  MR/MI  informs  both  pre- 
and  post-award  processes  and  decisions,  and  therefore  has  a  direct,  lasting  impact  on  the 
quality  of  the  product  or  service  the  government  receives  and  the  price  it  pays.  The  primary 
purposes  of  MR/MI  are  to  arm  the  acquisition  team  with  an  accurate  picture  of  the  state  of 
industry,  to  help  assess  the  feasibility  of  varying  procurement  options,  to  optimize  value  and 
mitigate  costs,  to  identify  potential  sources  of  supply  and  services,  to  identify  and  mitigate 
risks,  and  to  be  cognizant  of  similar  historical  procurements. 

A  handful  of  guides  and  tools  to  aid  in  the  conduct  of  market  research  exist,  but  they 
are  lacking  in  one  or  more  respects — they  are  either  vague  or  lacking  sufficient  detail  or 
examples,  more  prescriptive  than  descriptive,  too  lengthy — and  therefore  not  used  and  often 
ignored  by  the  majority  of  acquisition  professionals.  In  recognition  of  these  weaknesses,  the 
Naval  Postgraduate  School  recently  published  the  most  comprehensive  market  research 
guide  to  date  (Hawkins  et  al. ,  2012). 

Furthermore,  government  acquisition  personnel  tend  to  follow  a  “needs-based” 
archetype  for  market  research.  The  acquisition  team  first  determines  the  need  by  working 
with  the  user  to  refine  the  definition  of  the  requirement  to  come  to  a  common  understanding 
in  a  process  known  as  “requirements  definition,”  and  then  cross-checks  the  need  against 
existing  sources  of  supplies  or  contracts,  vendor  lists,  and  previous  purchases,  as  well  as 
consulting  with  the  small  business  office  as  applicable.  When  the  initial  market  research  is 
complete,  the  team  should  use  the  information  acquired  to  develop  the  acquisition  plan  and 
to  create  a  suitable  contract  structure  based  on  appropriate  evaluation  criteria  relevant  to 
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the  acquisition.  When  properly  applied,  market  research  is  a  powerful  pre-award  tool, 
although  market  research  should  not  stop  after  the  award  of  a  contract. 

Market  research  is  an  iterative  process  and  should  be  applied  over  the  entire  life 
cycle  of  an  acquisition.  Rather  than  a  reactive  stance  to  MR/MI,  a  more  optimal  solution 
involves  a  continual,  proactive  approach  that  yields  better  contracts  and  more  fluent  contract 
administration,  and  that  provides  acquisition  teams  the  leverage  they  need  to  obtain  the  best 
value  for  the  government.  To  obtain  the  benefits  of  MR/MI,  a  shift  in  the  current  culture  of 
acquisition  professionals  is  required. 

Historically,  anecdotal  evidence  shows  that  far  too  often,  market  research  is 
underscored  by  limited  effort  and  documentation  to  comply  with  the  general  requirement  to 
conduct  it  as  mandated  by  the  FAR,  which  results  in  another  box  to  check  on  a  lengthy  list 
of  mandated  pre-award  tasks.  Fully  realized,  MR/MI  can  better  inform  critical  acquisition 
processes  (see  Figure  1)  such  that  the  government  realizes  meaningful  differences  in 
needed  outcomes.  This  leads  to  the  following  recommendations. 

Recommendations 

To  become  proficient  at  gathering,  disseminating,  and  responding  to  market 
intelligence,  greater  attention  is  needed.  Currently,  market  research  is  a  stepchild  in  federal 
acquisition;  it  is  not  resourced  commensurate  with  its  importance  in  affecting  contracted 
needs.  Therefore,  we  offer  a  short  list  of  ideas  to  enable  a  stronger  infusion  of  market 
intelligence  into  outcome-driven  acquisition  decisions. 

•  Create  a  central  repository  of  market  reports  and  information  searchable  by 
NAICS  code  and  by  date.  This  will  help  acquisition  teams  share  gained 
knowledge  and  prevent  the  duplication  of  effort.  The  Air  Force  had  an  on-line 
market  research  repository  system  known  as  MRPost.  MRPost  was  a  good 
idea,  but  it  was  not  utilized  because  (1)  policy  did  not  enforce  usage,  (2)  it 
was  not  publicized  well  enough  to  users,  or  (3)  the  users  viewed  it  as  just 
another  task  to  perform  instead  of  a  valuable  source  of  information. 

•  As  Handfield  (2006)  recommended,  stand  up  a  central  market  intelligence 
cell  staffed  with  experts  in  certain  industries  who  are  available  to  generate 
market  analyses  to  acquisition  teams. 

•  Budget  for  market  intelligence,  such  as  that  found  in  syndicated  and 
customized  market  reports  (e.g.,  Gartner  Group,  Hoovers,  Dun  and 
Bradstreet  Supplier  reports,  IBISWorld,  and  the  Sourcing  Interest  Group). 

•  Develop  a  course  available  from  the  Defense  Acquisition  University  that 
teaches  best  practices  in  market  research  by  walking  the  students  through  a 
case  study  where  market  intelligence  made  the  difference  in  efficiency  and 
effective  contractor  performance. 
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Weapons  Acquisition  Reform:  Reform  Act  Is  Helping  DoD 
Acquisition  Programs  Reduce  Risk,  but  Implementation 

Challenges  Remain 


Michael  J.  Sullivan — Sullivan  is  the  director,  Acquisition  and  Sourcing  Management,  U.S. 
Government  Accountability  Office.  This  group  has  responsibility  for  examining  the  effectiveness  of  the 
DoD’s  acquisition  and  procurement  practices  in  meeting  its  mission  performance  objectives  and 
requirements.  In  addition  to  directing  reviews  of  major  weapon  system  acquisitions  such  as  the  Joint 
Strike  Fighter,  F-22,  Global  Hawk,  and  various  other  major  weapon  acquisition  programs,  Sullivan 
has  developed  and  directs  a  body  of  work  examining  how  the  DoD  can  apply  best  practices  to  the 
nation's  largest  and  most  technically  advanced  weapon  systems  acquisition  system.  This  work  has 
spanned  a  broad  range  of  issues  critical  to  the  successful  delivery  of  systems,  including  technology 
development,  product  development,  transition  to  production,  software  development,  program 
management,  requirement-setting,  cost  estimating,  and  strategic  portfolio  management.  The  findings 
and  recommendations  from  this  work  have  played  a  major  role  in  the  department’s  recent  acquisition 
policy  revisions.  Most  recently,  he  has  directed  the  GAO’s  annual  assessment  of  major  weapon 
systems  programs  for  the  Congress  and  GAO’s  work  with  Congress  in  establishing  acquisition  policy 
reforms.  His  team  also  provides  the  Congress  with  early  warning  on  technical  and  management 
challenges  facing  these  investments.  Sullivan  has  been  with  the  GAO  for  24  years.  He  received  a 
bachelor's  degree  in  political  science  from  Indiana  University  and  a  master’s  degree  in  public 
administration  from  the  School  of  Public  and  Environmental  Affairs,  Indiana  University. 
[sullivanm@gao.gov] 

Introduction 

For  several  decades,  the  GAO  has  reported  on  poor  outcomes  encompassing  cost 
and  schedule  growth  on  the  Department  of  Defense’s  (DoD’s)  major  weapon  acquisition 
programs.  Many  problems  can  be  traced  to  a  culture  where  the  military  services  begin 
programs  with  inflexible  requirements,  immature  technologies,  and  overly  optimistic  cost  and 
schedule  estimates.  Given  pressures  to  reduce  spending  across  the  government,  including 
the  DoD,  finding  ways  to  prevent  or  mitigate  cost  growth  is  crucial  to  U.S.  national  security. 

A  solid  program  foundation  using  good  developmental  testing  and  systems  engineering,  and 
reliable  cost  estimates  is  needed  in  order  to  help  avoid  cost  overruns  and  promote  better 
acquisition  outcomes.  There  have  been  numerous  attempts  in  the  past  to  improve  DoD 
acquisition  outcomes,  including  the  Packard  Commission  (President’s  Blue  Ribbon 
Commission  on  Defense  Management,  1986),  the  Goldwater-Nichols  Act  in  the  1980s 
(1986),  and  the  Federal  Acquisition  Streamlining  Act  of  1994.  More  recently,  Congress 
passed  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  (Reform  Act)1  to  improve  the 
way  weapon  systems  are  acquired  and  avoid  cost  and  schedule  overruns. 

In  2009,  the  Senate  Armed  Services  Committee  asked  us  to  begin  reporting  on  the 
DoD’s  implementation  of  Reform  Act  provisions  and  the  impact  the  Reform  Act  has  had  on 
weapon  acquisition  programs.  This  is  our  third  report  addressing  these  topics.  The  first 
report  focused  on  the  DoD’s  initial  efforts  to  implement  Reform  Act  provisions  for  systems 
engineering  and  developmental  testing,  including  the  placement  of  new  offices  for  these 
activities  within  the  Office  of  the  Secretary  of  Defense  (OSD;  GAO,  2010).  Our  second 
report  examined  the  challenges  the  Services  face  as  they  try  to  strengthen  systems 
engineering  and  developmental  testing  activities  on  weapon  acquisition  programs  (GAO, 


1  As  amended  by  the  Ike  Skelton  National  Defense  Authorization  Act  for  Fiscal  Year  201 1 ,  Pub.  L.  No. 
1 11-383  §§  813  and  1075,  and  the  National  Defense  Authorization  Act  for  Fiscal  Year  2012,  Pub.  L. 
No.  112-81  §§819  and  837. 
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2011b).  This  report  examines  (1)  the  DoD’s  progress  in  implementing  Reform  Act 
provisions;  (2)  the  impact  the  Reform  Act  has  had  on  specific  acquisition  programs;  and  (3) 
challenges  remaining  in  improving  the  weapons  acquisition  process. 

We  conducted  this  performance  audit  from  January  2012  to  December  2012  in 
accordance  with  generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate  evidence  to 
provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 
We  believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 

Background 

In  May  2009,  Congress  passed  the  Reform  Act  in  an  effort  to  improve  the  way 
weapon  systems  are  acquired  and  avoid  further  cost  overruns  on  such  programs.  When 
signing  the  Reform  Act  into  law,  the  President  stated  that  its  purpose  was  to  limit  weapon 
system  cost  overruns  and  that  it  would  strengthen  oversight  and  accountability  by  appointing 
officials  who  will  closely  monitor  the  weapons  systems  acquisition  process  to  ensure  that 
costs  are  controlled. 

Four  offices  were  established  as  a  result  of  the  Reform  Act:  SE,  DT&E,  CAPE,  and 
PARCA.  The  SE  and  CAPE  offices  existed  under  other  organizational  titles  prior  to  the 
Reform  Act.  Staffing  levels,  following  the  Reform  Act,  remained  relatively  stable  for  both  of 
these  offices,  but  some  reorganization  was  necessary  to  reflect  new  Reform  Act 
responsibilities.  The  DT&E  and  PARCA  offices  were  newly  established.  The  key  roles  and 
responsibilities  of  these  four  offices  as  outlined  in  the  Reform  Act  are  explained  in  Table  1 . 
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Table  1.  Key  Responsibilities  of  Offices  Established  as  a  Result  of  the 

Reform  Act 


Office _ Primary  responsibilities _ 

Systems  Engineering  •  serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under 

Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  on  systems 
engineering  activities  in  the  department 

develops  systems  engineering  and  development  planning  guidance  for  the 
DoD 

reviews  and  approves  major  defense  acquisition  program  systems  engineering 
plans 

monitors  major  defense  acquisition  program  systems  engineering  activities 

reports  to  Congress  annually  on  systems  engineering  organization, 
capabilities,  and  activities 

serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  on 
developmental  test  and  evaluation  activities 
develops  developmental  test  and  evaluation  guidance  for  the  DoD 
reviews  and  approves  major  defense  acquisition  program  developmental  test 
and  evaluation  plans 

monitors  and  reviews  acquisition  program  developmental  test  and  evaluation 
activities  of  major  defense  acquisition  programs 
reports  to  Congress  annually  on  developmental  test  and  evaluation 
organization,  capabilities,  and  activities 

Cost  Assessment  and  •  serves  as  principal  advisor  to  the  Secretary  of  Defense  and  other  senior 
Program  Evaluation  officials  on  matters  related  to  cost  analysis  and  the  planning  and  programming 

phases  of  the  planning,  programming,  budgeting,  and  execution  system 

•  develops  independent  cost  estimates  for  major  defense  acquisition  programs 
prior  to  major  milestone  decisions  and  updates  independent  cost  estimates 
when  programs  have  exceeded  critical  cost  thresholds,  known  as  Nunn- 
McCurdy  breaches 

reviews  existing  systems  and  methods  for  tracking  and  assessing  operation 
and  support  costs  on  major  defense  acquisition  programs 

develops  analysis  of  alternative  study  guidance  for  major  defense  acquisition 
programs 

approves  the  analysis  of  alternatives  study  plan  for  each  major  defense 
acquisition  program 

assesses  major  acquisition  program  performance  through  independent 
analyses  and  through  the  Defense  Acquisition  Executive  Summary  process 
identifies  the  root  causes  of  cost  growth  and  other  problems  on  programs  that 
experience  a  critical  Nunn-McCurdy  cost  breach 

Note.  This  table  was  created  using  GAO  analysis  of  the  Weapon  Systems  Acquisition  Reform  Act  of 
2009. 


Performance 
Assessments  and  Root 
Cause  Analyses 


Developmental  Test 
and  Evaluation 


In  addition  to  the  new  organizational  requirements,  the  Reform  Act  requires  the  DoD 
to  ensure  that  the  acquisition  strategy  for  major  defense  acquisition  programs  includes 
measures  to  ensure  competition  or  the  option  of  competition  throughout  the  program  life 
cycle.  This  could  include  strategies  such  as  maintaining  two  sources  for  a  system  (dual¬ 
sourcing)  and  breaking  requirements  for  supplies  or  services  previously  provided  or 
performed  under  a  single  contract  into  separate  smaller  contracts  (unbundling  of  contracts; 
Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  202).  Major  defense  acquisition 
programs  are  also  required  to  provide  for  competitive  prototyping — where  two  or  more 
competing  teams  produce  prototypes  before  a  design  is  selected  for  further  development — 
prior  to  Milestone  B  unless  a  waiver  is  properly  granted  by  the  milestone  decision  authority 
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(Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  202(a)),2  and  to  meet  the  following 
Milestone  B  certification  requirements,  including:3 

•  Appropriate  trade-offs  among  cost,  schedule,  and  performance  objectives 
have  been  made  to  ensure  the  program  is  affordable; 

•  A  preliminary  design  review  and  formal  post-preliminary  design  review 
assessment  have  been  conducted  and  on  the  basis  of  such  assessment  the 
program  demonstrates  a  high  likelihood  of  accomplishing  its  intended 
mission; 

•  Technology  has  been  demonstrated  in  a  relevant  environment  on  the  basis  of 
an  independent  review  and  assessment  by  the  Assistant  Secretary  of 
Defense  for  Research  and  Engineering; 

•  Reasonable  cost  and  schedule  estimates  have  been  developed  to  execute, 
with  concurrence  of  the  Director  of  CAPE,  the  program’s  product 
development  and  production  plan; 

•  Funding  is  available  to  execute  the  program’s  product  development  and 
production  plan; 

•  The  DoD  has  completed  an  analysis  of  alternatives  with  respect  to  the 
program;  and 

•  The  Joint  Requirements  Oversight  Council4  has  approved  program 
requirements,  including  an  analysis  of  the  operational  requirements. 

The  Reform  Act  also  requires  the  Joint  Requirements  Oversight  Council  to  ensure 
trade-offs  among  cost,  schedule,  and  performance  objectives  are  considered  for  joint 
military  requirements  (Weapon  Systems  Acquisition  Reform  Act  of  2009,  §  201).  The  GAO 
previously  reported  that  the  Council  considered  trade-offs  made  by  the  military  services 
before  validating  requirements,  but  the  military  services  did  not  consistently  provide  high- 
quality  resource  estimates  to  the  Council  for  proposed  programs  in  fiscal  year  2010.  We  also 
found  that  the  Council  did  not  prioritize  requirements,  consider  redundancies  across 
proposed  programs,  or  prioritize  and  analyze  capability  gaps  in  a  consistent  manner  (GAO, 
2011a). 


2  Specifically,  the  Reform  Act  required  the  DoD  to  modify  its  guidance  relating  to  the  operation  of  its 
acquisition  system  to  incorporate  these  competitive  prototyping  provisions.  The  DoD  did  so  through 
Directive-Type  Memorandum  (DTM)  09-027,  Implementation  of  Weapon  System  Acquisition  Reform 
Act  of  2009  (Dec.  4,  2009,  incorporating  Change  3,  Dec.  9,  2011).  Major  defense  acquisition 
programs  are  those  estimated  by  the  DoD  to  require  an  eventual  total  expenditure  for  research, 
development,  test,  and  evaluation,  including  all  planned  increments,  of  more  than  $365  million,  or  for 
procurement,  including  all  planned  increments,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant 
dollars  or  designated  by  the  USD(AT&L).  The  Milestone  Decision  Authority  for  major  defense 
acquisition  programs  is  the  USD(AT&L),  the  head  of  a  DoD  component,  or  if  delegated  the 
component  acquisition  executive. 

3  Pub.  L.  No.  1 1 1-23;  various  sections,  codified  at  10  U.S.C.  §  2366b.  The  Reform  Act  revised  the 
Milestone  B  certification  requirements  for  trade-offs,  preliminary  design,  technology  demonstration, 
and  reasonable  cost  and  schedule  estimates.  The  remaining  Milestone  B  certification  requirements 
bulleted  here  were  unrevised  by  the  Reform  Act. 

4  The  Joint  Requirements  Oversight  Council  is  an  advisory  council  to  the  Chairman  of  the  Joint  Chiefs 
of  Staff  with  the  responsibility  to:  (1)  identify,  assess,  and  approve  joint  military  requirements;  (2) 
assist  acquisition  officials  in  identifying  alternatives  to  acquisition  programs;  and  (3)  assist  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  assigning  priority  for  joint  military  requirements. 
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Findings 

The  GAO’s  analysis  of  11  weapon  acquisition  programs  showed  the  Reform  Act  has 
reinforced  early  attention  to  requirements,  cost  and  schedule  estimates,  testing,  and 
reliability.  For  example,  prior  to  starting  development,  an  independent  review  team  raised 
concerns  about  the  Ground  Combat  Vehicle  program’s  many  requirements  and  the  risks 
associated  with  its  seven-year  schedule.  Subsequently,  the  Army  reduced  the  number  of 
requirements  by  about  25%  and  prioritized  them,  giving  contractors  more  flexibility  in 
designing  solutions.  In  addition,  the  developmental  test  and  evaluation  office — resulting  from 
the  Reform  Act — used  test  results  to  help  the  Joint  Light  Tactical  Vehicle  program  develop  a 
more  realistic  reliability  goal  and  a  better  approach  to  reach  it.  Shown  in  Table  2  are  areas 
where  the  Reform  Act  influenced  several  programs  in  the  GAO’s  review. 


Table  2.  Reform  Act  Influence  on  Case  Study  Programs 


Program 

Requirements 

Cost  and  schedule 

Testing 

Reliability 

Before  Milestone  B 

Ground  Combat  Vehicle 

V 

V 

V 

✓ 

Joint  Light  Tactical  Vehicle3 

s 

V 

✓ 

Ohio  Class  Replacement 

✓ 

s 

✓ 

Ship  to  Shore  Connector3 

✓ 

✓ 

After  Milestone  B 

Joint  Strike  Fighter 

V 

Global  Hawk 

V 

✓ 

Gray  Eagle 

✓ 

V 

V 

✓ 

KC-46  Tanker 

s 

✓ 

Littoral  Combat  Ship  Seaframe 

s 

Remote  Minehunting  System 

✓ 

V 

✓ 

Small  Diameter  Bomb  II 

✓ 

s 

✓ 

Notes.  This  table  was  created  using  GAO  analysis  of  DoD  data. 

a  During  the  course  of  our  review,  the  Joint  Light  Tactical  Vehicle  and  Ship  to  Shore  Connector 
programs  held  a  Milestone  B  review. 


While  the  DoD  has  taken  steps  to  implement  most  of  the  fundamental  Reform  Act 
provisions,  some  key  efforts  to  date  have  been  primarily  focused  on  the  DoD’s  largest 
weapon  acquisition  programs.  The  DoD  faces  five  challenges — organizational  capability 
constraints,  the  need  for  additional  guidance  on  cost  estimating  and  Reform  Act 
implementation,  the  uncertainty  about  the  sufficiency  of  systems  engineering  and 
developmental  testing  resources,  limited  dissemination  of  lessons  learned,  and  cultural 
barriers  between  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the  military  services — 
that  limit  its  ability  to  broaden  the  Reform  Act’s  influence  to  more  programs.  Service  officials 
believe  additional  guidance  is  needed  to  improve  their  cost  estimates  and  other 
implementation  efforts.  They  also  believe  that  lessons  learned  from  programs  that 
experience  significant  cost  and  schedule  increases  should  be  shared  more  broadly  within 
the  acquisition  community.  These  challenges  seem  straightforward  to  address,  but  they  may 
require  more  resources,  which  have  been  difficult  to  obtain.  Ensuring  the  services  have  key 
leaders  and  staff  dedicated  to  systems  engineering  and  developmental  testing  activities, 
such  as  chief  engineers  at  the  service  level  and  technical  leads  on  programs,  as  well  as 
breaking  down  cultural  barriers  are  more  difficult  to  address.  They  will  require  continued 
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monitoring  and  attention  by  the  USD(AT&L),  service  acquisition  executives,  and  offices 
established  as  a  result  of  the  Reform  Act  to  address. 

Recommendations 

The  GAO  recommends  the  DoD  develop  additional  cost  estimating  and  Reform  Act 
implementation  guidance;  make  lessons  learned  available  to  the  acquisition  community;  and 
assess  the  adequacy  of  the  military  services’  systems  engineering  and  developmental 
testing  workforce.  The  DoD  generally  concurred  with  the  recommendations.  The  GAO 
clarified  one  recommendation  to  make  it  clear  that  the  DoD  needs  to  designate  an  office(s) 
within  the  Acquisition,  Technology,  and  Logistics  organization  to  provide  practical  Reform 
Act  implementation  guidance  to  program  offices. 

For  a  more  detailed  discussion  of  our  findings,  as  well  as  our  scope  and 
methodology,  see  www.gao.gov/products/gao-1 3-1 03. 
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Abstract 

To  increase  combat  effectiveness  by  networking  the  warfighter  and  to  easily  modify  and 
expand  its  existing  network  architecture,  the  United  States  Navy  requires  shipboard  computer 
systems  that  are  network-centric  and  service-based  and  that  support  open  architectures. 
However,  they  are  limited  by  the  radio  frequency  bandwidth  that  is  available  for  shipboard 
communications.  As  a  result,  some  network  applications  must  take  priority  over  others.  The 
current  Navy  prioritization  scheme  was  not  designed  with  the  needs  of  the  warfighter  as  the 
primary  focus  nor  does  it  allow  for  dynamically  changing  priorities  based  on  changing  threats. 
A  prioritization  scheme  is  proposed  that  optimizes  network  performance  based  on  warfighter 
needs.  The  scheme  is  developed  using  the  Capabilities-Based  Competency  Assessment 
process  introduced  by  Suttie  &  Potter  (2008)  applied  to  an  air  detect-to-engage  scenario  for  a 
carrier  strike  group  underway.  A  comparison  is  made  between  the  proposed  prioritization 
scheme  and  traditional  Navy  schemes  using  simulation.  Results  show  our  prioritization 
scheme  consistently  reduced  latency  and  increased  throughput  for  mission  relevant 
applications.  These  improvements  translate  directly  to  more  relevant  information  getting  to 
decision-makers  sooner,  which  leads  to  “information  superiority,”  ultimately  enhancing 
warfighting  capability. 

Introduction 

The  Program  Executive  Office  for  Command,  Control,  Communications,  Computers, 
and  Intelligence  (PEO  C4I)  Masterplan  summarizes  the  major  programs  of  the  Department 
of  the  Navy  (DoN)  applicable  to  network  operations,  providing  outlines  of  planned  future 
capabilities,  their  major  characteristics,  and  timelines  for  their  implementation.  It  includes  a 
mandate  for  fielded  computer  systems  to  be  network-centric,  service-based,  and  support 
open  architectures.  This  will  allow  the  Navy  to  field  a  rapid,  adaptable  warfighting  network, 
easily  tailored  to  the  task  at  hand  which  will  increase  combat  effectiveness.  Implementation 
of  this  capability  is  limited  by  the  network  resources — specifically  radio  frequency  (RF) 
communications  bandwidth — which  the  Navy  has  at  its  disposal  (PEO  C4I,  2011).  This 
means  some  network  applications  must  take  priority  over  others. 

To  understand  the  needs  of  the  warfighter,  this  study  looks  at  the  centerpiece  of  U.S. 
naval  strategy,  the  carrier  strike  group  (CSG).  Naval  carriers  are  dynamic  platforms 
equipped  with  a  wide  variety  of  systems  which  may  be  used  for  both  combat  and  non¬ 
combat  missions.  The  carrier  is  escorted  by  vessels  equipped  with  sensors  and  weapons 
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designed  for  battle  at  sea,  each  of  them  manned  by  technically  proficient  crews  capable  of 
not  only  naval  combat  but  also  disaster  and  humanitarian  relief.  Given  the  ability  to  reach 
distant  locations  in  a  timely  manner  and  its  operational  flexibility,  the  CSG  often  provides  the 
first  American  response  to  natural  disasters  both  in  the  U.S.  and  abroad.  As  the  central 
instrument  of  war  and  peace  for  the  Navy,  the  CSG  is  an  ideal  place  to  start  thinking  about  a 
prioritization  scheme  focused  on  the  warfighter. 

The  Navy  manages  network  Quality  of  Service  (QoS)  using  the  Automated  Digital 
Network  System  (ADNS).  The  current  network  prioritization  scheme  implemented  on  ADNS 
is  designed  to  optimize  network  performance  based  on  application  characteristics  and  does 
not  rank  applications  based  on  their  use  by  the  warfighter  in  a  combat  environment.  While 
this  approach  may  work  for  a  bandwidth-rich  environment  typically  found  in  the  civilian 
sector,  it  does  not  fully  support  the  main  purpose  of  Navy  tactical  networks,  that  is, 
warfighting. 

In  this  study,  we  use  the  Capability-Based  Competency  Assessment  (CBCA) 
suggested  by  Suttie  and  Potter  (2008)  to  link  CSG  air  detect-to-engage  mission  essential 
task  lists  (METLs)  to  the  personnel  and  systems  required  to  complete  them.  These 
competencies  act  as  operational  nodes  on  which  the  high-level  architecture  is  developed 
using  the  Department  of  Defense  Architectural  Framework  (DoDAF)  Version  1.5  products  to 
capture  the  roles  and  responsibilities  of  each  of  the  individuals  who  make  up  a  ship’s  air 
defense  team. 

The  resulting  prioritization  scheme  aligns  operational  nodes  and  services  within  the 
overall  system  architecture  so  that  commanders  are  able  to  more  effectively  use  existing 
network  resources  to  accomplish  required  tasks  within  a  compressed  time  frame.  By  linking 
the  identified  systems  to  the  application  types  ADNS  recognizes,  we  provide  mission- 
specific  justification  for  the  prioritization  of  one  network  application  over  another.  Finally,  we 
develop  a  simulation  model  that  captures  the  current  Navy  data  processing  environment. 
The  model  is  used  to  compare  our  prioritization  scheme  to  current  network  prioritization 
templates  in  the  context  of  an  air  detect-to-engage  scenario. 

This  study  illustrates  the  use  of  an  architectural  model  to  align  warfare  commander’s 
priority  and  intent  with  existing  network  capabilities  and  provides  a  common  tool  for 
communicating  warfare  commander’s  intent  to  those  responsible  for  carrying  out  that  intent. 
This  approach  should  be  used  to  help  Navy  networks  achieve  the  warfighting  capacity  for 
which  they  were  designed. 

Current  Bandwidth  Allocation  Scheme 

Given  the  different  roles  and  missions  that  the  CSG  is  expected  to  support,  flexibility 
in  communications  priorities  is  important.  As  air  operations  move  from  providing  defense 
capability  to  enabling  the  movement  of  supplies  and  evacuation  of  the  wounded,  network 
priorities  must  be  able  to  shift.  This  idea  extends  logically  to  varying  tactical  missions  as 
well.  The  priorities  during  air  defense  operations  are  not  the  same  as  those  during  an  anti¬ 
submarine  scenario  or  even  normal  underway  steaming.  Clearly,  the  overall  effectiveness  of 
the  CSG  will  be  maximized  by  giving  priority  to  mission-critical  applications,  which  change 
as  the  mission  changes. 

The  idea  of  mission-based  network  prioritization  has  not  been  lost  on  the  fleet  at 
large.  There  is  an  increased  demand  for  the  ability  to  modify  QoS  priorities,  based  on 
mission-specific  tasking  (Rambo,  2011).  The  goal  is  to  reduce  network  response  times  and 
increase  network  throughput  of  the  mission-critical  information,  thus  providing  more  time  for 
the  commander  to  make  the  “right”  choices,  leading  to  increased  mission  effectiveness  and 
less  wasted  network  resources. 
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The  Navy  currently  uses  the  Automated  Digital  Network  System  (ADNS)  to  allocate 
bandwidth  at  sea.  Initially  fielded  in  the  late  1990s,  ADNS  works  by  routing  outbound  data 
from  the  ship  through  the  various  radio  frequency  (RF)  paths  available  for  its  transmission. 
One  of  the  important  capabilities  of  ADNS  is  the  delivery  of  basic  QoS  capability.  QoS 
enables  the  network  to  make  “smart”  decisions  when  available  network  resources  are 
overtaxed  by  the  amount  of  information  they  are  being  required  to  route  (Rambo,  2011). 

ADNS  has  evolved  over  the  years  to  improve  bandwidth  management  and  enhance 
QoS  administration;  however,  there  is  still  room  for  improvement.  The  current  ADNS 
version,  Increment  Three  (ADNS  INC  III),  enables  QoS  through  static  application 
prioritization.  ADNS  works  to  mark  data  packets  generated  by  these  applications  and  then 
transmits  them  through  a  “packetshaper”  that  assigns  a  priority  to  the  traffic  being 
transmitted.  These  packets  are  then  sorted  into  bins  according  to  their  assigned 
prioritization  and  transmitted  accordingly.  The  prioritization  scheme  is  determined  by  the 
Naval  Cyber  Forces  (NCF)  command  and  can  only  be  modified  through  an  extended 
process  which  is  not  subject  to  change  by  ship’s  force  (Rambo,  201 1 ). 

Shipboard  networks  are  divided  into  Top  Secret/Sensitive  Compartmentalized 
Information  (TS/SCI),  Secret,  Unclassified,  and  separate  Coalition  classification  enclaves. 
There  is  an  additional  enclave  dedicated  to  network  overhead  and  encryption.  A  “type  of 
service”  header  is  assigned  within  each  classification  enclave  to  route  data  packets 
generated  by  shipboard  applications  to  various  network  queues.  Each  queue  is  allocated  a 
minimum  amount  of  bandwidth. 

Once  data  packets  have  been  routed  to  their  appropriate  queues,  transmission  is 
dictated  by  either  First  In  First  Out  (FIFO) — that  is,  the  first  data  packet  to  arrive  is  the  first  to 
leave — or  by  Cisco  Weighted  Random  Early  Detection  (WRED).  WRED  works  by  having  the 
network  router  (ADNS  in  this  case)  randomly  drop  IP  packets  being  sent  by  applications. 

The  dropping  of  packets  signals  that  the  network  is  congested,  causing  the  applications  that 
are  generating  the  packets  to  slow  down  the  rate  of  transmission.  Although  the  dropping  of 
packets  is  random,  the  probability  of  a  drop  is  not.  Applications  assigned  a  higher  priority 
have  a  lower  probability  of  drop  and  thus,  a  higher  throughput.  Additionally,  if  applications 
are  not  utilizing  the  minimum  bandwidth  allowed,  that  bandwidth  is  shared  with  other 
applications. 

The  prioritization  in  ADNS  is  done  via  a  formal  submission  process  and  the 
application  priority  is  validated  by  Naval  Cyber  Forces  (Rambo,  2011).  Given  the  changing 
priorities  of  separate  mission  areas,  it  is  imperative  that  shipboard  personnel  be  able  to 
assign  prioritizations  dynamically  to  shipboard  network  services.  This  need  continues  to 
grow  as  the  Navy’s  Consolidated  Afloat  Networks  and  Enterprise  Services  (CANES)  system 
is  fielded. 

CANES  will  serve  to  consolidate  and  replace  five  existing  legacy  networks  afloat. 
These  systems  include  the  Integrated  Shipboard  Network  System  (ISNS),  Sensitive 
Compartmented  Information  (SCI)  Networks,  and  Combined  Enterprise  Regional  Information 
Exchange  System  Maritime  (CENTRIXS-M).  Using  the  Service-Oriented  Architecture 
(SOA)1  concept,  CANES  will  eliminate  redundant  legacy  hardware  and  replace  them  with  a 
single,  consolidated  system.  According  to  the  CNO’s  CANES  Initial  Implementation  and 
Action  Message,  DTG  071927Z  DEC  09,  all  shipboard  systems  that  will  be  fielded  after  the 
implementation  of  CANES  must  be  compatible  with  the  new  common  network  hardware. 


1  Lund  et  al.  (2007)  defined  Service-Oriented  Architecture  (SOA)  in  the  military  context  as  “a  way  of 
making  military  resources  available  as  services  so  they  can  be  discovered  and  used  by  other  entities 
that  need  not  be  aware  of  those  services  in  advance.” 
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This  single,  common  computing  environment  will  provide  the  necessary  framework  to 
implement  QoS  at  a  higher  level  of  granularity. 

The  Capabilities-Based  Competency  Approach 

The  Capabilities-based  Competency  Assessment  (CBCA)  was  developed  at  the 
Naval  War  College  for  manpower  analysis.  It  seeks  to  identify  functional  roles  working  within 
a  team  construct  versus  looking  at  billets  and  shipboard  occupations.  Functional  roles  are 
linked  to  “subtasks”  which  together  define  the  complete  mission-level  tasking.  The  major 
distinction  of  CBCA  is  the  focus  on  capability  versus  a  set  of  competencies  (Suttie  &  Potter, 
2008).  Once  the  capability  inherent  to  the  role  is  understood,  its  relationship  to  other  roles 
working  in  the  total  system  can  be  comprehended. 

Unlike  the  traditional,  billet-based  allocation  of  personnel,  CBCA  links  METLs  to  the 
personnel  and  systems  required  to  complete  them.  It  defines  “roles”  which  act  as  critical 
nodes  that  correspond  to  a  DoDAF  Operational  Node  Connectivity  Description  (OV-2; 

Suttie,  2011)  of  the  overall  operational  architecture.  These  roles  are  capability  based  and 
independent  of  the  personnel  assigned  to  complete  them. 

This  study  uses  the  CBCA  approach  by  first  identifying  METLs  related  to  a  CSG  air 
detect-to-engage  scenario.  The  METLs  are  then  used  to  describe  a  set  of  competencies 
including  operations,  personnel,  and  system  requirements  inherent  to  air  defense 
operations.  The  Service-Oriented  Architecture  framework  is  formed  by  assigning  METLs  to 
the  operational  nodes  responsible  for  their  execution.  These  relationships  can  be  captured 
in  a  DoDAF  Operational  Activity  Model  Description  (OV-5).  This  model  is  completed  in 
conjunction  with  a  DoDAF  Systems  Functionality  Description  (SV-4),  which  not  only 
captures  the  decomposition  of  the  top-level  activity,  but  also  identifies  the  systems  used  to 
enable  functionality.  Finally,  the  relationships  between  the  operators,  their  responsible 
actions,  and  the  systems  used  to  complete  those  actions  are  captured  via  a  DoDAF 
Operational  Activity  to  Systems  Function  Traceability  Matrix  (SV-5a).  By  doing  so,  the 
relationships  between  the  operational  nodes  and  the  systems  that  each  node  uses  to 
accomplish  those  tasks  are  identified. 

These  products  are  used  to  understand  the  relationships  between  operator  and 
machine  and  allow  the  warfare  commanders  to  assign  the  correct  prioritization  to  the 
systems  at  their  disposal.  Once  form  has  been  matched  to  function,  it  is  possible  to 
understand  which  nodes  and,  as  a  result,  which  systems  are  needed  to  complete  an 
aggregate  task.  This  process  provides  justification  and  realization  of  the  most  beneficial 
arrangement  for  network  prioritization.  By  assigning  the  highest  level  of  prioritization  to 
those  network  applications  needed  to  accomplish  mission-appropriate  tasking,  a  strike 
group’s  network  resources  are  used  to  their  fullest  capability.  The  performance  of  all  other 
systems  that  are  not  crucial  to  the  completion  of  the  assigned  tasking  should  be  sacrificed  in 
order  to  benefit  those  that  are  imperative. 

Defining  the  Operational  Nodes 

Before  system  prioritization  can  take  place,  the  users  that  will  operate  the  system 
must  be  identified.  For  the  CSG  air  detect-to-engage  scenario,  this  is  accomplished  using  a 
DoDAF  OV-2  diagram  showing  the  relationships  between  a  single  air-defense  unit  (ADU) 
and  the  off-ship  warfare  commanders  and  coordinators  (see  Horton,  2012,  p.  23).  The  next 
step  is  to  identify  the  tasks  associated  with  each  user  related  to  air  defense  operations. 
These  tasks  can  be  found  in  the  Navy’s  Universal  Naval  Task  List  (UNTL)  discussed  in  the 
next  paragraph. 


m  khsiikm 
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The  UNTL  describes  tasks  that  can  be  completed  by  naval  forces.  The  UNTL  is  used 
by  commanders  to  determine  what  tasks  can  be  accomplished  by  the  naval  elements  under 
their  commands.  METLs  are  derived  from  this  list  and  are  used  to  support  a  commander’s 
assigned  mission.  They  serve  as  a  command’s  list  of  tasks  that  are  considered  essential  for 
mission  accomplishment  (Chief  of  Naval  Operations,  Commandant,  U.S.  Marine  Corp, 
Commandant,  U.S.  Coast  Guard,  2007).  The  UNTL  is  subdivided  into  separate  task  levels 
for  each  level  of  warfare.  The  prefix  for  tactical  level  tasks  is  TA,  thus  naval  tasks  at  the 
tactical  level  are  known  as  Navy  Tactical  Tasks  (NTA).  An  examination  of  the  UNTL  reveals 
which  NTA’s  are  relevant  to  air  defense.  By  using  the  descriptions  provided  in  the  UNTL  for 
each  NTA,  it  is  possible  to  compile  a  list  of  those  tasks  which  are  related  to  air  defense  (see 
Horton,  2012,  p.  34). 

A  DoDAF  OV-5  describes  the  operations  required  to  complete  a  mission  and  shows 
the  flow  between  operational  activities.  The  model  is  constructed  by  taking  each  of  the  NTAs 
identified  as  relevant  to  air  defense  operations,  establishing  a  hierarchy  of  those  tasks,  and 
mapping  each  NTA  to  the  operational  node  responsible  for  its  completion  (see  Horton,  2012, 
p.  36). 

Having  identified  the  operational  activities  involved  in  the  process  of  conducting  air 
defense  and  linking  each  of  these  activities  to  the  operational  node  responsible  for  their 
completion,  the  next  step  is  to  identify  the  information  systems  that  each  of  those 
operational  nodes  require  to  complete  their  assigned  tasking.  Linking  the  form  to  function 
will  provide  the  justification  for  our  prioritization  scheme. 

Identifying  Systems  Required  for  Air  Defense  Operations 

The  Command,  Control,  Communications,  Computers,  and  Intelligence  (C4I) 
Masterplan  serves  to  summarize  the  major  attributes  of  DoN  network-centric  systems.  The 
Masterplan  provides  C4I  system  baselines  for  each  type  of  ship,  including  carriers  and  ships 
assigned  to  the  CSG.  These  baseline  descriptions  may  be  used  to  identify  systems  which 
communicate  via  ADNS.  By  using  the  system  descriptions  presented  in  the  C4I  Masterplan, 
a  list  was  developed  of  those  systems  required  to  conduct  air  defense  operations  (Table  1). 

It  should  be  noted  that  while  the  systems  chosen  provide  a  good  representative 
sample  of  those  systems  which  may  be  used  in  air-defense  operations,  this  list  should  by  no 
means  be  considered  exhaustive.  The  C4I  Masterplan  provides  only  system  overviews  and 
does  not  give  detailed  explanations  of  each  system  and  its  capabilities.  In  order  to  correctly 
identify  each  relevant  system,  subject  matter  experts  on  each  would  need  to  be  consulted, 
and  personnel  familiar  with  the  entire  C4I  portfolio  would  need  to  compile  an  exhaustive  list. 
For  purposes  of  this  study,  however,  it  is  sufficient  to  include  these  systems  to  illustrate  our 
approach. 
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Table  1.  Air  Defense  Net-Centric  Systems 

(adapted  from  PEO  C4I,  2011) 


System  Name 

Description 

Ship  Type 

Ship’s  Signal 
Exploitation 
Equipment 
(SSEE) 

Increment  E/F 

Provides: 

1)  Direction  finding  (DF) 

2)  Signal  acquisition 

3)  Hostile  Forces  Integrated  Targeting  Service  (HITS) 

4)  Digital  Receiver  Technology  (DRT)  geolocation 
capability 

5)  Integrated  signal  analysis  and  select  National  Security 
Agency  (NSA)  applications  via  the  Cryptologic  Unified 

Build  (CUB)  toolbox 

CVN,  CG,  DDG 

AN/USQ- 
172(V)10  Global 
Command  and 
Control  System- 
Maritime  (GCCS- 
M) 

Provides: 

1)  Unit  location  and  amplifying  information 

2)  Fuses,  correlates,  filters,  maintains,  and  displays 
location  and  attribute  information  on  friendly,  hostile,  and 
neutral  land,  sea,  and  airforces,  integrated  with  available 
intelligence  and  environmental  information  to  develop 
Common  Operational  Picture  (COP) 

3)  Aides  decision-maker 

CVN,  CG,  DDG 

Distributed 
Common  Ground 
System-Navy 
(DCGS-N) 

Provides: 

1)  Integrates  shared  intelligence  data,  information,  and 
services  between  various  intelligence  and  decision¬ 
making  entities 

2)  Distributable  intelligence  products 

CVN 

Naval  Integrated 
Tactical 
Environment 
System,  Variant 

IV  (NITES-IV) 

Provides: 

1)  Operational  and  tactical  METOC  support  to  Navy, 

Marine  Corps,  and  Joint  Forces  engaged  in  worldwide 
operations,  ashore  and  afloat 

2)  Distributes  gathered  meteorological  data 

CVN 

Using  these  systems,  we  can  capture  the  capabilities  each  one  provides.  This  is 
accomplished  using  a  System  Functionality  Description.  The  DoD  (2007)  guidance  in 
Architecture  Framework,  Version  1.5,  Volume  II,  defines  a  System  Functionality  Description 
(SV-4a)  as  documenting  system  functional  hierarchies  and  system  functions  and  how  data 
flows  between  them.  A  System  Functionality  Description  for  air  defense  is  constructed  by 
taking  each  of  the  systems  identified  as  relevant  to  air  defense  operations  and  breaking 
them  down  to  their  required  functionality.  The  relationships  between  those  systems  are  then 
mapped,  providing  the  structure  of  the  viewpoint  (see  Figure  1). 

Flaving  now  identified  the  functionality  that  each  air-defense  unit  provides,  we  are 
ready  to  link  the  system  function  to  the  operational  tasks  we  previously  identified.  This  is 
accomplished  using  a  Systems  Functional  Traceability  Matrix,  as  described  in  the  next 
section. 


1  HKH.’IVT 
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Figure  1.  Conduct  Air  Defense  (DoDAF  SV-4a)  System  Functionality  Description 
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Linking  Operational  Activities  to  Systems  Functions 

A  Systems  Functional  Traceability  Matrix  (DoDAF  SV-5a)  documents  the  relationship 
between  the  operational  activities  and  system  functionality  present  in  the  overall  architecture 
(see  Table  2).  Those  systems  which  are  being  used  by  an  operator  to  complete  a  task  are 
indicated  with  an  X  in  Table  2.  For  now,  only  those  systems  that  connect  to  the  Global 
Information  Grid  (GIG)  via  an  Internet  Protocol  (IP)  pipeline  have  been  mapped.  As  new 
systems  are  fielded  to  be  deployed  on  CANES,  this  diagram  would  need  to  grow  to  include 
them.  The  dashed  area  indicates  that  those  systems  are  not  currently  available  for  those 
users. 


By  identifying  the  systems  used  by  operators  to  complete  assigned  tasks,  it  is 
possible  to  identify  the  systems  most  useful  to  a  mission,  in  this  case,  air  operations.  These 
are  the  systems  which  should  be  given  priority  in  an  air  detect-to-engage  scenario.  This 
methodology  can  be  applied  to  any  given  mission  or  tasking. 

Each  information  system  has  now  been  linked  to  the  task  associated  with  its  use, 
and  each  task  has  been  linked  to  the  operator  who  completes  that  task.  Our  proposed 
prioritization  scheme  will  place  each  of  the  identified  systems  at  the  top  of  the  priority 
scheme.  A  comparison  of  the  current  priority  scheme  and  our  proposal  will  be  outlined  in  the 
next  section. 
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Table  2 


Conduct  Air  Defense  SV-5a,  Systems  Function  Traceability  Matrix 
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Quality  of  Service  Model 

Quality  of  Service  (QoS)  management  for  shipboard  IP  networks  is  implemented  by 
marking  IP  packets  using  the  “type  of  service”  (ToS)  header  field.  The  Automated  Digital 
Network  System  (ADNS)  uses  the  first  six  bits  within  the  header  to  mark  each  packet  with  a 
Differentiated  Services  Code  Point  (DSCP;  Automated  Digital  Network  System,  2011). 
These  DSCP  markings  can  be  used  to  separate  network  traffic  into  class  bins  which  can  be 
used  to  implement  separate  controls  in  off-ship  transmission.  The  routing  of  packets  is  done 
without  regard  for  the  security  level  classification. 

To  test  the  effectiveness  of  a  prioritization  scheme  in  the  current  Navy  environment, 
we  need  to  model  the  DSCP  process  used  by  ADNS.  A  stochastic  simulation  was 
developed  using  the  ExtendSim  8  software  to  model  this  process.  Figure  2  shows  the  basic 
outline  of  the  model  that  will  be  used  to  aid  discussion  of  QoS  implementation  within  ADNS. 
It  is  important  to  note  that  our  simulation  focuses  on  how  prioritization  schemes  impact  data 
throughput  and  latency  within  the  context  of  the  air  detect-to-engage  scenario.  We  are  not 
modeling  the  events  that  might  occur  in  the  scenario,  but  rather  using  the  scenario  to 
understand  the  expected  information  requirements  and  data  traffic  within  each  phase  of  an 
air  detect-to-engage  (DTE)  scenario. 

ADNS  separates  network  traffic  into  five  separate  Community  of  Interest  (COI)  local 
area  networks  (LANs).  They  are  SECRET,  TS-SCI,  UNCLASS,  CENTRIXS  (coalition),  and 
an  additional  classification  for  Cipher  Text  Core  Traffic  (Automated  Digital  Network  System, 
201 1 )  and  are  shown  on  the  left  side  of  Figure  2.  Each  LAN  is  comprised  of  various  IP- 
based  network  applications  which  are  classified  according  to  queuing  doctrine,  such  as 
First-In,  First-Out  (FIFO)  or  Class-Based  Weighted  Fair  Queuing  (CBWFQ).  These 
applications  are  listed  within  the  Traffic  Classes,  Packet  Marking,  and  Priority  Processing 
documentation  provided  by  the  Program  Manager,  Warfare  (PMW)  160  Office. 
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Figure  2.  Flow  Diagram  Representation  of  ExtendSim  8  Model 
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Each  of  the  applications  which  comprise  the  COI  LANs  is  represented  in  our  model 
by  a  block  that  creates  “packets”  at  a  random  interval.  Mean  inter-arrival  time  for  each  type 
of  application  varies  depending  on  the  type  of  service  it  performs,  as  shown  in  Table  3. 


Table  3.  Application  Type  Inter-Arrival  Parameters 


Application  Type 

Mean  Inter-Arrival  Period 

Standard 

Deviation 

Video 

33  ms 

1  ms 

VoIP 

100  ms 

10  ms 

Data 

200  ms 

20  ms 

Network  Overhead 

50  ms 

1  ms 

The  inter-arrival  periods  were  modeled  using  a  normal  distribution,  bounded  by  zero 
on  the  left  side,  with  a  standard  deviation,  as  indicated  in  Table  3.  Although  network  traffic 
behavior  is  usually  “bursty”  and  the  inter-arrival  times  are  not  typically  normally  distributed, 
we  chose  the  normal  distribution  for  simplicity.  In  addition,  we  use  a  “worst  case”  scenario  in 
which  every  application  is  creating  the  maximum  amount  of  data  possible,  with  1,500  bytes 
per  packet.  While  the  two  simplifying  assumptions  introduced  in  our  model  would  most  likely 
not  occur  in  real-life,  they  facilitate  comparison  of  prioritization  schemes  and  limit  the 
number  of  independent  variables  in  the  model. 

Each  of  the  packets  generated  in  the  simulation  was  marked  with  a  priority  based 
upon  the  type  of  information  it  is  carrying.  This  marking  allows  for  the  packet  to  be  routed  to 
one  of  the  fourteen  separate  queues,  as  shown  in  Figure  2.  ADNS  currently  specifies  13 
different  queue  types,  based  upon  network  application  behavior  (Automated  Digital  Network 
System,  2011).  We  introduce  a  14th  Mission  Queue  which  is  reserved  for  those  applications 
deemed  most  relevant  to  air  defense  operations  based  on  our  previous  analysis.  This  is  the 
simplest  way  to  test  our  proposed  prioritization  scheme  against  the  existing  ADNS  scheme. 
Actual  implementation  of  the  prioritization  scheme  by  the  Navy  might  differ  based  on 
network  configuration  and  other  considerations. 

The  model  is  designed  to  incorporate  only  those  bandwidth  pipelines  available  to  a 
particular  class  of  ship.  Thus,  CVNs  will  be  allowed  the  CWSP,  SHF,  and  EHF  pipelines, 
and  DDGs  and  CGs  will  be  allowed  the  SHF,  EHF,  and  INMARSAT  pipelines.  The  model 
works  to  balance  the  load  between  each  of  the  transmission  pipelines  available  to  each 
queue  type  shown  in  Figure  2. 

The  model  checks  each  time  step  to  see  which  queues  require  bandwidth  and  which 
do  not.  It  will  first  subtract  that  amount  of  bandwidth  that  has  been  assigned  to  the  queues 
that  currently  require  it  from  the  total  amount  of  bandwidth  available.  Then  it  will  parse  out 
the  remaining  bandwidth  following  the  same  percentage  assignment  schedule  as  outlined  in 
the  Traffic  Classes,  Packet  Marking,  and  Priority  Processing  documentation  provided  by  the 
PMW  160  Office. 

ADNS  uses  two  methods  for  the  queuing  doctrine  applied  to  each  queue.  First, 
applications  which  are  weighted  equally  within  the  same  queue  are  handled  by  a  FIFO 
methodology.  Second,  applications  which  are  weighted  differently,  though  routed  to  the 
same  queue,  are  handled  using  CBWFQ  with  Weighted  Random  Early  Detection  (WRED). 
CBWFQ  will  route  those  packets  with  a  higher  priority  at  the  expense  of  those  with  a  lower 
priority.  This  is  accomplished  by  randomly  dropping  lower  priority  packets  once  a  queue  has 
reached  a  pre-determined  length.  In  our  model,  we  sample  the  current  queue  length  for 
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each  time  step.  If  the  sampled  queue  length  falls  within  the  set  boundaries,  packets  are 
dropped  according  to  scheduled  packet  drop  probability. 

Within  ADNS,  random  dropping  denies  the  originating  application  a  receipt 
acknowledgment  and  forces  the  application  to  retransmit  the  packet.  Eventually,  this  causes 
the  originating  application  to  slow  down  its  transmission  rate,  allowing  higher  priority 
applications  to  transmit  at  a  faster  rate  (Automated  Digital  Network  System,  2011).  In  our 
model,  this  metric  is  captured  by  measuring  the  amount  of  packets  that  actually  were 
transmitted  and  comparing  that  value  to  the  amount  of  packets  that  were  created.  This  gives 
a  percentage  of  actual  throughput  and  will  be  used  as  a  measure  to  compare  the 
effectiveness  of  a  given  priority  scheme  as  it  applies  to  mission-specific  applications. 

Results  and  Conclusion 

The  simulation  model  was  designed  to  measure  latency  and  throughput.  Latency 
refers  to  the  timeliness  of  data.  By  recording  latency,  we  gain  an  understanding  of  how  long 
it  takes  for  data  to  be  created,  routed,  and  then  transmitted.  Throughput  refers  to  how  much 
of  the  data  created  is  actually  transmitted  in  the  time  allowed.  Throughput  is  an  indicator  of 
the  quality  of  the  transmission. 

The  air  detect-to-engage  scenario  consists  of  three  stages — surveillance,  escalation, 
and  terminal.  During  the  surveillance  phase,  there  is  no  threat  and  normal  air  defense 
operations  are  in  effect.  The  surveillance  phase  provided  baseline  measurements  of  latency 
and  throughput  using  current  ADNS  settings.  Average  percent  throughput  and  latency  for 
both  the  carrier  (CVN)  and  the  cruiser/destroyer  (CRUDES)  escorts  over  a  total  of  30  runs 
were  recorded. 

Next  we  modeled  the  escalation  phase.  During  this  phase,  the  strike  group  receives 
indications  of  a  pending  attack  on  the  high  value  unit  (HVU).  In  response,  the  strike  group 
commander  will  probably  increase  the  threat  warning  posture  which  brings  the  force  to  a 
higher  state  of  readiness  in  preparation  for  a  possible  attack  via  the  air.  To  support  this 
condition,  we  propose  the  prioritization  scheme  shown  in  Table  4  because  it  prioritizes  the 
systems  designed  to  aid  anti-air  warfare. 

The  bandwidth  percentages  assigned  to  each  queue  are  intended  to  minimize 
latency  and  maximize  throughput  of  systems  relevant  to  air  defense,  while  minimizing  the 
impact  to  other  systems.  It  should  be  noted  that  these  percentages  are  notional,  but  should 
be  selected  so  that  they  support  the  information  needs  of  the  commander. 
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Table  4. 


CBCA  Bandwidth  Allocation  Scheme — Escalation  Phase 


I sc  ala  tun  Phase 

CWSP 

SHF 

EHF 

INMARSAT 

Croup  1 

33% 

19% 

N/A 

N/A 

CEM 

15% 

25% 

N/A 

N/A 

VIC 

12% 

12% 

N/A 

NA 

JCA 

18% 

12% 

N/A 

NA 

SECRET  (CBWFQl) 

7% 

27% 

:■% 

UNCLAS  (CBWFQ2) 

0% 

4% 

12% 

7% 

CENTONS  (CBWFQ3) 

5% 

4% 

12% 

4% 

SCKCSWFOt) 

13% 

5% 

17% 

14% 

Otktr 

Vol?  (LLQ) 

3$414pi 

3S4i±>p» 

NA 

57 14  pi 

FOtFMV) 

10% 

10% 

N/A 

NA 

UDP 

NA 

KA 

10% 

NA 

LSSOCOM 

24% 

24% 

N/A 

NA 

CTXtt  (CONTROL) 

1% 

1% 

2% 

2% 

0AMD«auS 

11% 

25% 

5% 

41% 

Miiion 

21% 

21% 

15% 

15% 

Table  4  shows  the  queues  currently  utilized  with  the  ADNS  (Automated  Digital 
Network  System,  2011)  as  well  as  a  Mission  queue  that  implements  our  prioritization 
scheme.  The  four  columns  presented  in  Table  6  represent  the  four  transmission  paths 
available  to  the  strike  group  ships:  Commercial  Wideband  Satellite  Program  (CWSP) — CVN 
only,  Super  High  Frequency  (SHF),  Extremely  High  Frequency  (EHF),  and  International 
Maritime  Satellite  (INMARSAT) — CRUDES  only  (Automated  Digital  Network  System,  2011). 
The  values  in  each  block  represent  the  percentage  of  bandwidth  available  on  each 
transmission  path,  that  is,  column,  applied  to  each  queue,  that  is,  row,  with  the  exception  of 
Voice  over  Internet  Protocol  (VoIP),  which  is  a  flat  amount. 

In  the  escalation  phase,  we  assume  that  the  traffic  output  of  systems  relevant  to  air 
defense  would  increase  due  to  the  now-present  threat  and  the  information  being  gathered 
about  it.  For  modeling  purposes,  we  doubled  the  data  output  in  this  phase.  The  average 
latency  (milliseconds)  and  throughput  (percentage)  over  30  runs  was  recorded  and 
compared  with  latency  and  throughput  for  each  data  type  for  each  prioritization  scheme. 

The  third  phase  of  evaluation  is  the  terminal  phase.  During  this  phase,  the  inbound 
threat  has  fired  its  weapon  at  the  HVU,  prompting  the  commander  to  further  escalate  the 
strike  group’s  readiness  posture.  To  support  this  condition  of  readiness,  we  propose  the 
prioritization  scheme  shown  in  Table  5.  The  bandwidth  percentages  selected  for  this  phase 
reflect  an  increased  amount  of  air-defense  relevant  network  traffic.  Again,  percentages  are 
notional.  Actual  percentages  would  be  based  on  the  commander’s  priority  and  intent. 
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Table  5. 


CBCA  Bandwidth  Allocation  Scheme — Terminal  Phase 


T  erninal  Phase 

CUSP 

sir 

EHF 

INMARSAT 

Group  1 

30% 

15% 

N/A 

NA 

CEM 

15% 

25*/. 

N/A 

NA 

VTC 

12% 

12% 

N/A 

NA 

JCA 

IS*/. 

12% 

N/A 

NA 

SECRET  (CBWFQl) 

12*/. 

7% 

25% 

15% 

UN'CLAS  (CBWFQ2) 

8% 

4% 

11% 

7% 

CEXTREiS  (CBWFQ3) 

6% 

4% 

10“/. 

4*/. 

SCI  (CBWFQl) 

13% 

5% 

15% 

13% 

Other 

VoPCLLQ) 

3S41d>pj 

3S41d>ps 

N/A 

57  lips 

FO(FMV> 

10% 

10% 

N/A 

NA 

LDP 

NA 

N/A 

10"/. 

NA 

L5SOCOM 

22% 

22*/. 

N/A 

NA 

CTXet  (CONTROL) 

1% 

1% 

2% 

2% 

QAM  Default 

10% 

25% 

5% 

37% 

Mission 

27% 

27% 

22*/. 

22% 

The  data  output  of  the  air  defense  applications  was  again  effectively  doubled — now 
four  times  the  initial  value,  assuming  that  the  traffic  output  of  those  applications  would 
increase  significantly  during  the  terminal  phase. 

Independent  two-sample,  single-tailed  Student  t-tests  were  conducted  to  determine 
whether  there  is  a  statistically  significant  difference  between  the  baseline  latency  and 
throughput  and  the  latency  and  throughput  using  our  prioritization  scheme.  The  results  are 
shown  in  Tables  6-9. 

Table  6.  CARRIER  Latency  Hypothesis  Test  Results 


CARRKR  LATENCY  HYPOTHESIS  TEST 

APPLICATION  NAME 

PHASE 

H. 

H. 

tc* 

Reject  H0 

Wgn  Priority 

A so  cat  on 

Esa  ae  on 

M 

1* 

II 

X*  >  A*  2 

153 .023 

1672 

Yes 

Terminal 

A\  =  A', 

A\  >  A*  j 

95.038 

1672 

Yes 

GCCS-M,  NETPREC 

Esca  at  on 

A;=XS 

Xz  >  Xz 

8A214 

1672 

Ves 

Terminal 

X  =T. 

X;>X . 

123 .36- 

1672 

Yes 

Time  Sync. Chat. 

Cos.  HP  Of 

Esca  at  on 

x-.=xz 

As  >  A' 2 

162-315 

1672 

Yes 

Terminal 

A.  =  A* « 

X .  >  A*  2 

3a  383 

1672 

Yes 

Email.  CERCtS. 

OS/BD,  P4RA126. 

Esca  at  on 

A\  =  A". 

Xj  >  Xz 

128-046 

1672 

Vo 

Terminal 

A*.  =  A, 

X-.  >Xj 

29.111 

1672 

Y« 

Name  Resolution, 
Encryption.  Pie 

Esca  at  on 

X;=*2 

X.  >x. 

190009 

1672 

Yes 

Terminal 

X,  =  X2 

Xl  >  Xz 

27.310 

1672 

Yes 

EVCP.  BgB-ot“ier. 
tSKT 

Esa  at  on 

X-.=*2 

A.  >  A' j 

210,061 

1672 

Yes 

Terminal 

x.  =  A*  2 

A\  >A. 

26.628 

1672 

Yes 
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Table  7. 


CRUDES  Latency  Hypothesis  Test  Results 


CRUDES  LATENCY  HYPOTHESIS  TEST 

APPLICATION  NAME 

PHASE 

H. 

H. 

N* 

Reject  H  a 

Wfh  Priority 

Aoo'  cat  ons 

tsa  at  on 

X,  =  Aj 

A..  >  X, 

101 .763 

1672 

Ves 

Terminal 

fU  =  X, 

A,>A, 

56.621 

1672 

Ves 

GCCS-  Nl  NETPREC 

Esa  at  on 

A.  =  A, 

C  >A: 

89  S3 

1672 

Vo 

Terminal 

H 

IN 

II 

A.  >A. 

59  29: 

1672 

Ves 

Ti  me  Sync,  Chat. 

Coo.  HP  OP 

Esa  at  on 

A;  =  A. 

H 

H 

A 

•  ** 

* 

173 .293 

1672 

Vo 

Terminal 

A*.  =  A* j 

A*2  >  A*  2 

55.064 

1672 

Ves 

E-rjJ.  CERCtS. 

OS/RD.  P AAA  126. 

Esa  at  on 

A.  =  A; 

A:  >  Aj 

174.210 

1672 

Vo 

Terminal 

y 

n 

Hi 

M 

A-  >A3 

57.352 

1672 

Ves 

Nam  e  Resort  on. 
Encryption,  Pile 

Esa  at  on 

*=Xj 

A’.  >  A*  2 

157.239 

1672 

Vo 

Terminal 

A*.  —  A* j 

A.  >  A"  2 

6D  639 

1672 

Yes 

EVCP.  Bg9rother, 
tSRT 

Esa  at  on 

A:=X: 

A.  >X. 

161 954 

1672 

Vo 

Terminal 

A.  =  A* 2 

A-  >  A*  2 

51898 

1672 

Yes 

Table  8.  CARRIER  Throughput  Hypothesis  Test  Results 


CARRIER  THROUGHPUT  HYPOTHESIS  TEST 

APPLICATION  NAME 

PHASE 

H. 

H. 

tm .» 

Ni« 

Reject  H0 

W51  Priority 

Esa  at  on 

X.  -  A. 

As  <  X  2 

1015 

-1.672 

No 

Aoo  cat  ons 

Termina' 

A:  =  A  . 

A.  <  A*  2 

-1.172 

-1.672 

No 

GCCS-M,  NETPREC 

Esa  at  on 

A.  =  A, 

A*  <A*2 

0418 

-1972 

No 

Terminal 

As  =  A; 

A.<As 

-0.6B5 

-1972 

No 

Time  Sync. Chat. 

Esa  at  on 

A.  =  A, 

As  <  Ax 

*3-2.7 

-1972 

Yes 

Coo.  HP  DP 

Terminal 

X  =  A, 

As  <  X. 

-21COO 

-1.672 

Ves 

tma  .  CERCtS, 

Esa  at  on 

As  =  A  . 

Xs  <X: 

■1206 

-1972 

No 

06/ BO.  PAAA126. 

Terminal 

H 

II 

'4* 

X  <  A 2 

-22156 

-1972 

Ves 

Name  Resolution. 

Esa  at  on 

Xs  =  Xz 

H 

A 

H 

•3908 

-1972 

Vo 

Encryption,  Pile 

Terminal 

A.  =  A*  2 

A.  <  A  2 

•21693 

-1.672 

Ves 

EVCP.  B'f  Brother, 

Esa  at  on 

X  =  X. 

Aj  <  A. 

-2424 

-1972 

Vo 

ISRT 

Terminal 

h 

11 

K 

AC  <  X. 

-22553 

-1.672 

Ves 

Table  9.  CRUDES  Throughput  Hypothesis  Test  Results 


CRUDES  THROUGHPUT  HYPOTHESIS  TEST 

APPLICATION  NAME 

PHASE 

H. 

H. 

t*. 

Reject 

Wjh  Priority 

Esa  at  on 

As  —  A. 

A!  <  A  - 

8.063 

-1972 

No 

Ass  cat  ons 

Terminal 

As  =  A  2 

X\  <  x2 

-1514 

-1.672 

No 

GCCS-M,  NETPREC 

Esa  at  on 

A;  =  Aj 

As  <  Aj 

-9  £55 

-1972 

Ves 

Terminal 

A:  -  Xz 

Aj  <  As 

-1472 

-1£72 

No 

Ti  me  Sync.  Chat. 

Esa  at  on 

A;  =  Aj 

A’i  <  Xz 

-427218 

•1£72 

Yes 

Coo.  HP  DP 

Terminal 

X  =  A  2 

A.  <  A  2 

-158.074 

-1.672 

Ves 

Email,  CERCtS. 

Esa  at  on 

* 

II 

H 

As  <X: 

-415.372 

-1972 

Ves 

OS/BO.  PARA126. 

Terminal 

Hi 

II 

HI 

A.  <  A*  2 

-158343 

-1972 

Yes 

NameResoiutan. 

Esa  at  on 

H 

II 

As  <  Aj 

-337.051 

-1972 

Ves 

Encryption,  P  e 

Terminal 

Hi 

II 

Hi 

M 

A.  <  A  2 

-169.129 

-1.672 

Yes 

EVCP.  B  g  Brother. 

Esa  at  on 

x  =x. 

As  <  Aj 

-373909 

-1972 

Ves 

tSRT 

Terminal 

II 

H 

A.  <  As 

*16a739 

-1£72 

Yes 

We  note  that  there  is  a  statistically  significant  decrease  in  the  average  latency 
associated  with  each  of  the  selected  applications  using  our  prioritization  scheme  as 
compared  to  default  ADNS  settings.  Our  results  also  indicate  statistically  significant 
increases  in  throughput  using  our  prioritization  scheme  for  most  applications;  however,  there 
is  no  significant  difference  for  some  applications.  We  note  decreases  in  percent  throughput 
for  the  High  Priority  Applications  data  types  for  both  the  CARRIER-  and  CRUDES-type  ships 
during  the  Escalation  Phase  as  well  as  GCCS-M,  NETPREC  data  types  for  the  CARRIER 
during  the  Escalation  Phase  when  using  our  prioritization  scheme.  This  decrease  in  percent 
throughput  is  offset  by  marked  decreases  in  associated  latency  which  should  be  taken  into 
consideration  when  implementing  our  process  for  network  prioritization. 
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An  important  question  is  whether  the  differences  noted  in  Tables  6-9  are  practically 
significant.  One  of  the  primary  reasons  for  the  selection  of  the  air  detect-to-engage  scenario 
is  that  time  is  often  at  a  premium.  For  example,  consider  the  time  savings  for  the  CRUDES 
class  ships  during  the  terminal  phase  of  engagement.  Our  prioritization  scheme  saves  on 
average,  approximately  9s  in  time  delays  for  our  selected  applications  as  compared  to  the 
default  ADNS  prioritization  scheme.  In  order  to  understand  the  importance  of  this  time 
savings,  we  use  the  cruising  speed  of  a  typical  hostile  missile,  the  C-801  (595  knots).  Using 
the  formulas  for  time  distance,  we  see  the  actual  distance  the  missile  may  travel  in  this 
allotted  time  is  almost  one  and  a  half  nautical  miles. 


d  =vt 

d  =  (0. \65nmi/s)(9s)  ~  1 .486 nmi 

So  ultimately,  what  does  the  time/distance  savings  buy  us?  As  the  Navy  becomes 
more  and  more  net-centric,  more  shipboard  systems  will  be  used  in  the  identification  and 
prosecution  of  hostile  targets.  Every  millisecond  we  save  in  the  transmission  of  data  results 
in  increased  ranges  at  which  we  may  engage  hostile  targets.  This  means  more  time  for 
human  decision-makers  to  draw  conclusions  and  more  opportunities  for  us  to  put  ordnance 
on  target.  In  their  book,  Human  Factors  in  Simple  and  Complex  Systems,  Proctor  and  Van 
Zandt  (2008)  defined  a  reaction-time  task  as  that  which  requires  a  person  to  respond  to  a 
stimulus  as  quickly  as  possible.  They  highlighted  recent  work  conducted  in  continuous 
information  accumulation.  They  noted  that  the  fastest  possible  human  reaction  to  visual 
stimuli  is  150  ms.  This  reaction  time  slows  linearly,  following  a  log2  scale,  with  the  number  of 
possible  stimuli  and  responses  available  to  the  operator. 

If  we  assume  the  previously  described  mean  reaction  time,  we  see  that  the  time 
savings  described  in  this  paper  are  within  the  threshold  of  human  reaction.  This  is  critical  as 
it  allows  for  an  actual  physical  response  by  a  human  operator.  The  more  the  latency  of  our 
selected  data  is  reduced,  the  more  time  the  human  decision-maker  has  to  react  to  the  visual 
stimulus.  This  impact  is  even  more  pronounced  if  we  consider  the  near  instantaneous 
reaction  time  of  automated  systems.  Given  an  autonomous  response  capability, 
milliseconds  saved  in  transmission  time  can  directly  translate  to  whether  an  enemy  target 
may  be  destroyed  in  the  allotted  time  or  if  it  will  strike  its  intended  target. 

We  have  demonstrated  a  process  that  seeks  to  align  system  prioritization  with 
operator  needs  based  upon  mission  tasking.  We  accomplish  this  by  linking  operational 
tasking  to  warfighters  and  identifying  those  systems  used  by  the  warfighters  to  accomplish 
said  tasking.  Our  work  may  be  seen  as  a  guideline  for  the  development  of  network 
prioritization  schemes  which  seek  to  optimize  Navy  networks  for  combat  and  are  in  keeping 
with  the  philosophy  of  net-centric  warfare  (NOW).  Ideally,  this  approach  will  help  strike  group 
commanders  see  their  networks  as  true  weapon  systems  and  help  bring  to  the  forefront 
those  network  systems  relevant  to  the  mission-at-hand. 
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Abstract 

The  U.S.  Navy  transitioned  to  computer-based  training  (CBT)  in  A  and  C  schools  in  2003 
after  a  2001  Revolution  in  Training  report  claimed  that  the  Navy  would  realize  savings  in  cost 
and  training  time  without  negatively  affecting  the  quality  of  sailors  arriving  to  the  fleet. 
Anecdotal  evidence  from  ship  personnel  suggested  otherwise.  This  study  analyzes 
maintenance  data  for  the  AN/SGG-89(v)  sonar  system  to  determine  whether  the  transition  to 
CBT  contributed  to  increased  fleet  maintenance  costs. 

Government  studies  showed  that  the  conversion  to  CBT  was  not  the  sole  contributing  factor 
to  increased  fleet  maintenance  costs  or  degraded  fleet  material  readiness.  Changes  to  the 
Navy’s  training,  maintenance,  and  manning  programs  during  the  early  2000s  were  all 
contributing  factors.  If  the  conversion  to  CBT  were  to  have  an  effect  anywhere  in  the  Navy 
maintenance  system,  it  should  be  seen  in  maintenance  activities  where  sailors  were 
performing  maintenance  on  ships.  Our  analysis  revealed  that  the  average  cost  of  these 
activities  was  significantly  greater  after  CBT  was  implemented.  This  would  support  the 
anecdotal  evidence  that  CBT  was  impacting  the  quality  of  maintenance  on  ships. 

Introduction 

Traditionally,  the  majority  of  specialized  skills  training  (known  as  “A”  and  “C”  schools) 
in  the  Navy  has  taken  place  in  a  classroom  setting  with  instructors.  At  the  turn  of  the 
century,  Navy  leadership  became  concerned  that  current  training  programs  would  not 
adequately  meet  future  demands.  As  a  result,  the  chief  of  naval  operations  (CNO)  chartered 
an  Executive  Review  of  Navy  Training  (ERNT)  to  review  the  Navy  training  system  and 
recommend  solutions  to  improve  training  effectiveness  and  meet  future  training  demands. 

The  ERNT  group  noted  that  formal  schoolhouse  training  requires  a  large  investment 
in  facilities,  instructors,  and  laboratories  and  that  future  training  demand  would  outstrip  the 
number  of  billets  available  under  the  legacy  schoolhouse  system  (Executive  Review  of  Navy 
T raining  [ERNT],  2001 ).  They  suggested  that  the  use  of  new  training  technologies  could 
help  meet  that  demand  while  reducing  the  cost  of  training.  Motivated  by  these  findings,  the 
Navy  established  Task  Force  EXCEL  (Excellence  through  Commitment  to  Education  and 
Learning)  to  develop  a  continuum  of  lifelong  learning,  use  a  streamlined  funding  process 
and  a  single  training  authority,  create  a  Human  Performance  Systems  Model  (HPSM),  and 
link  training  and  acquisition  (Naval  Personnel  Development  Command,  2002). 
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Part  of  the  Navy’s  new  strategy  included  the  use  of  new  training  technologies  such 
as  distributed  learning,  computer-based  training  (CBT),  collaborative  learning,  and 
computer-mediated  learning.  The  Navy  claimed  that  the  introduction  of  CBT  would  reduce 
both  training  time  and  training  costs  without  reducing  the  quality  of  training  received  (ERNT, 
2001).  Accordingly,  CBT  was  introduced  full-time  into  the  training  pipeline  in  fiscal  year  (FY) 
2003. 


A  2009  Naval  Inspector  General  (IG)  Report,  Computer  Based  Training ,  reported 
that  the  introduction  of  CBT  did  reduce  training  time.  However,  sailors  arriving  to  the  fleet 
under  CBT  did  not  usually  meet  the  required  Knowledge,  Skills,  Abilities,  and  Tools  (KSATs) 
upon  reporting  on  board.  Because  of  this,  ships  had  to  take  the  time  to  train  sailors  up  to 
acceptable  standards  (Naval  Inspector  General,  2009).  This  suggests  that  while  initial 
training  costs  may  have  been  reduced  by  CBT,  the  overall  cost  of  operations  and 
maintenance,  including  on-the-job  training  (OJT),  may  have  increased. 

This  study  examines  the  impact  of  CBT  on  Navy  training  costs  as  well  as  operations 
and  maintenance  costs  before  and  after  the  implementation  of  CBT.  We  first  look  at 
Department  of  the  Navy  (DoN)  Budget  Reports  from  FY2000  through  FY2010  to  determine 
the  macro-level  impact  of  CBT  on  Navy  costs.  At  the  macro  level,  there  are  many  variables 
besides  CBT  that  could  contribute  to  changes  in  maintenance  costs,  including  the  Global 
War  on  Terror  (GWOT)  and  increased  operations  tempo  (OPTEMPO).  However,  it  is 
impractical  to  isolate  the  impact  of  CBT  on  Navy  maintenance  costs  at  the  macro  level. 
Instead,  it  is  necessary  to  look  at  the  impact  of  CBT  on  a  particular  system,  program,  or 
technology.  This  research  effort  focuses  on  a  single  system,  the  AN/SQQ-89(v)  sonar, 
collecting  data  at  a  level  of  detail  that  allows  for  the  control  of  the  various  variables  that 
might  impact  maintenance  costs. 

We  start  with  a  discussion  of  the  Navy’s  classroom  training  system,  the  Revolution  in 
Training  and  CBT,  followed  by  a  look  at  the  Navy  maintenance  process  and  changes  in 
manning  and  maintenance  policies  during  the  2000s.  Next,  we  focus  on  a  single  Navy 
system,  the  AN/SQQ-89(v)  sonar  system,  and  examine  how  the  conversion  to  CBT  might 
have  affected  maintenance  costs  in  that  system. 

Training 

Training  in  the  Navy  occurs  throughout  a  sailor’s  career.  After  completing  recruit 
training,  sailors  are  sent  to  specialized  skill  training  in  their  designated  job  specialty,  or 
rating.  In-rate  training  begins  in  A  school,  where  sailors  learn  the  particular  skills  specific  to 
their  job.  From  there,  a  sailor  can  receive  additional  training  in  C  school.  Once  a  sailor  is 
assigned  to  a  ship,  he  or  she  receives  training  for  collateral  duties  such  as  quarterdeck 
watches,  anti-terrorism/force  protection  watches,  weapons  handling,  and  the  at-sea  fire 
party.  Additionally,  sailors  can  expect  to  receive  general  military  training  in  topics  ranging 
from  electrical  safety  to  suicide  prevention. 

Traditional  Schoolhouse  Training 

Until  the  early  2000s,  in-rate  training  in  the  Navy  was  conducted  in  a  formal 
schoolhouse  setting,  where  instructors  delivering  the  training  are  subject  matter  experts 
(SMEs)  on  the  material  they  are  teaching  (ERNT,  2001).  Typically,  SMEs  come  from  the 
fleet  and  have  experience  working  on  the  equipment  they  are  teaching  about.  Training  is 
delivered  in  the  form  of  lectures,  and  instructors  are  able  to  supplement  the  lecture  material 
with  tips  and  anecdotes  from  their  career  experiences  (Naval  Inspector  General,  2009). 

In  addition  to  lectures,  sailors  can  reinforce  their  understanding  of  the  material 
through  hands-on  experience  in  a  laboratory  setting.  In  maintenance  courses,  students  are 
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able  to  work  on  the  exact  equipment  they  will  see  in  the  fleet,  and  instructors  are  able  to 
simulate  equipment  casualties  for  technicians  to  troubleshoot.  Instructors  are  able  to  tailor 
the  delivery  of  material  to  a  class  based  on  the  students’  levels  of  comprehension.  For 
example,  if  a  class  has  difficulty  understanding  a  particular  concept,  the  instructor  can 
choose  to  spend  more  time  in  the  lab  to  reinforce  what  is  learned  during  the  classroom 
portion. 

There  are  several  benefits  to  instructor-led  training  (ILT).  Since  a  single  instructor 
teaches  a  large  group  of  students,  group  learning  techniques  can  be  employed  that  would 
otherwise  be  unavailable  in  one-on-one  or  CBT  instruction.  The  formation  of  small  groups 
within  a  class  fosters  team-building  and  allows  students  to  help  and  teach  each  other. 
Compared  to  the  costs  of  software  development,  testing,  and  hardware  purchase,  ILT  is  in 
some  ways  more  cost  effective,  depending  on  class  size  and  length  of  use.  Additionally,  the 
controlled  classroom  environment  offers  fewer  distractions  than  CBT  or  distance  learning. 
Finally,  ILT  doesn’t  take  as  long  to  develop  as  CBT.  It  takes  approximately  34  hours  to 
develop  one  hour  of  ILT  (Chapman,  2007),  while  it  takes  approximately  220  hours  to 
develop  a  standard  e-learning  course  (Chapman,  2006). 

ILT  also  has  its  disadvantages.  Since  everyone  has  different  learning  capabilities, 
some  students  may  be  more  advanced  and  become  bored  while  waiting  for  slower  learners 
to  catch  up.  Conversely,  slow  learners  may  have  difficulty  keeping  up.  Depending  on  the 
size  and  duration  of  the  course,  ILT  may  be  more  expensive  than  CBT. 

Revolution  in  Training 

In  October  2000,  the  Executive  Review  of  Navy  Training  (ERNT)  group  was  charged 
with  providing  insights  on  how  to  improve  and  align  training  organizations,  leverage  civilian 
training  practices,  and  use  new  technologies  to  provide  a  continuum  of  training  for  sailors. 
The  24-member  group  was  comprised  of  military  and  civilian  personnel,  members  of 
academia,  research  institutions,  and  industry.  In  2001,  ERNT  released  their  report, 
Revolution  in  Training:  Executive  Review  of  Navy  Training  Final  Report. 

During  their  review,  the  ERNT  group  noted  that  the  demands  for  training  had 
increased.  At  the  macro  level,  the  training  demands  are  driven  by  the  Required  Operational 
Capabilities  and  Projected  Operating  Environments  (ROC/POE).  ROC/POE  is  a  tool  that  is 
used  to  determine  specific  warfighting  missions  for  each  ship.  Training  requirements  are 
derived  from  these  missions  and  are  then  used  to  determine  specific  training  requirements 
for  sailors. 

Changes  in  the  ROC/POE  lead  to  increased  ship  training  requirements  which  are 
passed  down  to  the  sailor  level.  The  ERNT  group  noted  that  the  finite  number  of  seats 
available  in  the  Navy  schoolhouses  was  not  able  to  support  the  increased  training  demands. 
Because  of  this,  there  were  gaps  in  the  types  of  training  that  current  and/or  potential  sailors 
needed  and  what  could  be  delivered. 

In  many  cases,  this  resulted  in  billets  which  could  not  be  filled  because  there  were  no 
sailors  with  the  required  training  to  fill  them.  During  the  1990s,  several  other  items 
contributed  to  the  lack  of  trained  sailors.  First,  the  pool  of  experienced  sailors  had 
decreased  due  to  drawdowns  and  retirements.  Second,  it  was  difficult  to  compete  for  trained 
personnel  in  a  healthy  U.S.  economy,  and  many  trained  sailors  were  leaving  for  jobs  in  the 
civilian  sector. 

The  ERNT  group  suggested  that  technology  and  the  science  of  learning  offered 
several  opportunities  to  improve  the  Navy  training  system  by  reducing  training  time  through 
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CBT  and  offering  distributed  learning  opportunities  that  could  be  executed  at  the  workplace. 
This  is  discussed  further  at  the  end  of  the  report. 

Computer-Based  Training 

Computer-based  training,  or  CBT,  is  defined  as  “individual  or  group  self-paced 
instruction  using  a  computer  as  the  primary  training  medium,  to  include  web-delivered  Navy 
E-Learning  (NEL)”  (Naval  Inspector  General,  2009,  p.  ii).  In  Navy  A  schools,  students  go 
through  learning  modules  on  a  personal  computer  at  their  own  pace.  When  students  are 
done  processing  the  information  presented  on  the  screen,  they  click  “next”  to  proceed  to  the 
next  piece  of  information.  There  are  usually  small  knowledge  assessments  throughout  the 
module,  followed  by  a  final  knowledge  assessment  at  the  end  of  the  module  (Naval 
Inspector  General,  2009,  p.  7). 

Because  the  learning  is  self-paced,  instructors  were  replaced  with  “facilitators.” 
Facilitators  are  not  necessarily  SMEs  in  the  subject  matter  being  delivered  in  the  CBT 
modules.  The  purpose  of  the  facilitator  is  “to  ensure  classroom  rules  are  followed,  assist  with 
computer-related  issues,  and  monitor  student  progress.  They  do  not  provide  reinforcement 
of  learning  objectives  or  enhance  retention  of  course  material.”  The  problem  with  replacing 
instructors  with  facilitators  is  that  students  cannot  go  to  a  facilitator  with  a  question  about 
subject  material,  removing  the  opportunity  to  teach  when  a  student  is  confused  (Naval 
Inspector  General,  2009). 

There  are  several  advantages  to  CBT.  The  learning  is  self-paced  and  if  the  course  is 
offered  as  distance  learning,  the  schedule  to  take  the  course  is  flexible.  Students  can 
complete  the  course  at  their  own  paces,  which  generally  shortens  training  time.  Since  there 
are  no  instructors  involved,  the  message  doesn’t  change  from  one  person  to  the  next 
(Dhanjal  &  Calis,  1999).  In  addition,  the  Navy  was  able  to  reduce  training  time  using  CBT, 
which  resulted  in  cost  savings  in  training  manpower  and  infrastructure,  as  noted  by  the  Navy 
IG  (2009)  and  the  GAO  (201 0). 

However,  the  use  of  CBT  raised  concerns  in  the  fleet  about  the  level  of  knowledge  of 
sailors  reporting  to  ships  from  A  schools.  The  inspector  general  (IG)  noted  that  sailors 
arriving  to  the  fleet  under  CBT  did  not  usually  meet  the  required  KSAT  standards  and  were 
unfamiliar  with  the  equipment  they  would  be  working  on  and  the  tools  they  would  need  to 
use.  Because  of  this,  ships  had  to  take  the  time  to  train  sailors  up  to  acceptable  standards. 

In  fleet  interviews,  some  commands  reported  that  qualification  time  was  nearly  double  what 
it  was  before  the  introduction  of  CBT  (Naval  Inspector  General,  2009).  The  GAO  reports  in 
2010  and  201 1  made  similar  observations  and  concluded  that  the  change  to  CBT  had  a 
negative  impact  on  readiness. 

The  Navy  IG  and  GAO  reports  found  that  while  the  Navy’s  use  of  CBT  resulted  in 
cost  and  training  time  savings,  the  quality  of  sailor  reporting  to  the  fleet  was  not  as  well 
prepared  as  ILT-trained  sailors  of  the  past.  The  result  is  that  poorly-trained  sailors  may  have 
contributed  to  declining  material  readiness  in  the  fleet.  The  next  section  of  this  study 
examines  Navy  maintenance  practices  and  highlights  the  findings  of  the  2010  Fleet  Review 
Panel  on  Surface  Force  Readiness  report. 

Maintenance 

Navy  maintenance  occurs  on  three  levels:  organizational  level  (O-level),  intermediate 
maintenance  (IM)  activities,  and  depot  level.  This  section  of  the  study  discusses  all  three 
maintenance  levels.  Additionally,  this  section  discusses  changes  made  to  the  maintenance 
process  in  2003  which  were  reported  on  in  the  2010  Fleet  Review  Panel  on  Surface  Force 
Readiness  (known  as  the  Balisle  Report  for  its  chairman,  Vice  Admiral  [VADM,  Retired] 
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Phillip  Balisle),  a  report  that  discussed  declining  fleet  readiness  as  a  result  of  changes  to 
training,  maintenance,  and  manning  policies  in  the  early  2000s. 

Shipboard  maintenance  begins  with  the  Planned  Maintenance  System  (PMS).  PMS 
is  governed  by  Naval  Sea  Systems  Command  (NAVSEA;  2003)  Instruction  4790. 8B,  Ship’s 
Maintenance  and  Material  Management  (3-M)  Manual.  The  instruction  outlines  the 
requirements  for  PMS  on  shipboard  systems  and  equipment.  The  purpose  of  PMS  is  to 
provide  ships  with  the  means  to  plan,  schedule,  and  perform  preventive  maintenance 
onboard  and  to  identify  potential  equipment  problems  before  the  equipment  fails. 

If  corrective  maintenance  is  required,  the  maintenance  is  reported,  scheduled,  and 
performed  through  O-level  shipboard  maintenance.  Ship  maintenance  actions  are  reported 
in  Navy  Visibility  and  Management  of  Operating  and  Support  Costs  (VAMOSC),  under  Unit 
Level  Consumption  and  Manhours — Organizational  Corrective  Maintenance. 

Intermediate  maintenance  (IM)  is  “normally  performed  by  Navy  personnel  onboard 
tenders,  repair  ships,  Shore  Intermediate  Maintenance  Activities  (SIMAs),  aircraft  carriers, 
and  fleet  support  bases”  (Naval  Sea  Systems  Command,  2003,  p.  1-5).  IM  jobs  are  deferred 
corrective  maintenance  jobs  that  are  beyond  the  capability  of  the  ship’s  force  and  are  sent 
off-ship  for  completion.  IM  is  tracked  in  Navy  VAMOSC  under  Maintenance — Intermediate. 

Depot-level  maintenance  “requires  major  overhaul  or  a  complete  rebuilding  of  parts, 
assemblies,  subassemblies,  and  end  items,  including  the  manufacturing  of  parts, 
modifications,  testing,  and  reclamation”  (Naval  Sea  Systems  Command,  2003,  p.  1-5).  Depot 
maintenance  is  reported  in  Navy  VAMOSC  under  Maintenance  and  Modernization — Depot, 
Other  Depot. 

In  2009,  VADM  (Ret.)  Phillip  Balisle  was  directed  to  conduct  a  Fleet  Review  Panel 
(FRP)  of  surface  force  material  readiness.  The  report  noted  that  4,052  billets  were  removed 
from  Navy  ships  from  2001-2009.  While  billets  were  removed  from  ships,  requirements  such 
as  maintenance,  damage  control  watches,  training,  and  in-port  duties  were  not  reduced 
(Balisle,  2010).  The  shortcomings  of  CBT  described  in  the  previous  section  exacerbated  the 
problems  experienced  with  manning  reductions  since  sailors  were  not  arriving  on  board  with 
the  right  KSATs.  The  result  was  undermanned  ships  with  poorly  trained  sailors  with  not 
enough  time  or  know-how  to  perform  routine  maintenance  actions. 

In  addition  to  reduced  fleet  manning,  shore  facilities  also  received  manning  cuts.  This 
means  that  maintenance  that  was  intended  for  intermediate  maintenance  activities  was 
pushed  back  to  ship  personnel,  which  were  undermanned  and  poorly  trained.  In  addition  to 
the  shrinking  shore  workforce,  the  amount  of  time  the  ships  were  available  was  shortened 
from  15  weeks  to  nine  weeks  (Balisle,  2010).  These  actions  resulted  in  equipment  being  out 
of  commission  for  longer  periods  of  time. 

Finally,  the  2010  Balisle  report  noted  that  changes  in  PMS  were  made  because  ships 
couldn’t  meet  maintenance  requirements  due  to  reduced  manning.  Maintenance 
requirements  were  either  eliminated  or  extended  in  periodicity.  The  intent  was  to  shift 
maintenance  requirements  to  shore  facilities,  but  since  manning  was  reduced  ashore,  many 
requirements  went  away  completely.  The  elimination  and  extension  of  maintenance 
requirements  can  lead  to  more  opportunities  for  equipment  to  become  inoperable,  resulting 
in  degraded  fleet  readiness  (Balisle,  2010). 

The  Navy  introduced  several  major  changes  to  training,  maintenance,  and  manning 
policies  during  the  early  part  of  the  2000-2010  decade.  The  Balisle  report  found  that  training 
was  a  factor,  but  certainly  not  the  only  factor,  that  led  to  degraded  fleet  readiness.  Manning 
reductions  would  have  led  to  cost  savings  in  the  military  personnel  budget,  but  the  impact  of 
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the  reductions  may  have  resulted  in  maintenance  cost  increases  in  future  budgets  due  to 
deferred  maintenance  actions,  thus  confounding  the  effect  of  CBT.  Similarly,  changes  in 
maintenance  policies  may  have  impacted  maintenance  costs  in  future  years.  At  a  macro 
level,  the  impact  of  CBT  is  impossible  to  tease  out  (see  Gibson,  2012,  for  an  examination  of 
Navy  training,  operations,  and  maintenance  budgets  between  2000  and  2012).  For  this 
reason,  we  decided  to  examine  one  system  in  particular,  the  AN/SQQ-89(v)  sonar  system, 
in  hopes  that  we  could  separate  the  two  factors. 

AN/SQQ-89(v)  Sonar  System 

To  examine  the  effect  of  CBT  on  rising  maintenance  costs,  this  study  will  focus  on 
the  operating  and  support  (O&S)  costs  of  a  single  Navy  system,  the  AN/SQQ-89(v)  sonar 
system,  and  look  at  how  the  conversion  to  CBT  affected  maintenance  costs  in  that  system. 
An  analysis  by  Gibson  (2012)  showed  that  manning  levels  for  sonar  technicians  did  not 
change  significantly  from  FY2000-FY2010,  effectively  eliminating  manning  as  a  contributor 
for  the  AN/SQQ-89  O&S  costs  and  focusing  the  study  on  training  and  maintenance. 

The  AN/SQQ-89(v)  surface  ship  Anti-Submarine  (ASW)  Warfare  combat  system 
(referred  to  as  “the  89”  in  the  rest  of  this  paper)  is  an  integrated  network  of  sonar  systems 
designed  to  search,  detect,  classify,  and  engage  ASW  threats.  The  system  is  currently 
installed  on  CG-47  class  cruisers,  DDG-51  class  destroyers,  and  FFG-7  class  frigates.  The 
89  uses  a  variety  of  sensors  that  can  transmit  (active)  and  receive  (passive)  acoustic  data  in 
order  to  detect  and  classify  threats.  Data  from  the  sensors  can  be  correlated  and  targets  can 
be  localized  using  Target  Motion  Analysis  (TMA)  to  generate  a  firing  solution  for  weapons 
systems  (Jane’s  Information  Group,  2010). 

The  89  system  consists  of  15  different  variants.  Variants  differ  based  on  the  sensors 
chosen  and  the  version  of  each  sensor.  In  this  report,  only  variants  2,  3,  4,  6,  7,  and  9  were 
studied.  These  variants  were  chosen  because  they  were  on  board  ships  prior  to  the 
introduction  of  CBT  into  the  sonar  training  pipeline  (2003)  and  remained  on  board  after  CBT 
was  introduced.  This  allows  for  analysis  of  ship-maintenance  trends  both  prior  to  and  after 
the  introduction  of  CBT.  A  list  of  ships  per  variant  is  given  in  Table  1 . 
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Table  1. 


List  of  Ships  and  AN/SQQ-89(v)  System  Variants  Used  in  This  Study 


(V)2 

SHIP 

HOMEPORT 

(v}3 

SH*> 

HOMEPORT 

(V)6 

SHIP 

HOMEPORT 

CG  55 

LEYTE  GULF 

fertik.  V A 

CG  56 

SAN  JACINTO 

Ncrtilc.  VA 

CG  65 

AN30 

Nortik.  VA 

FFGS 

MCMERNEY 

UaSCOTt  FL 

CG  57 

LANS  CHAMPLAIN 

Sar  Dego.  CA 

CG  69 

VICKSBURG 

Mavport  FL 

FFG  25 

BOCNE 

Mascot  FL 

CG  55 

PHIFPINE  SEA 

Mascot  R. 

CG  70 

LAKE  ERE 

Pear  Harbo.  H 

FFG  29 

STEPFENW  GROLES 

Mascot  FL 

CG  71 

CAFE  ST GEORGE 

Sar,  Dego.  CA 

FFG  32 

JOHN  HALL 

Mascot  FL 

(V)4 

SHI> 

HOMEPORT 

CG  72 

VELLA  GULF 

Nortik.  VA 

FFG  33 

JARFTET 

San  Dego.  CA 

DDG51 

AREIGHBUWCE 

Ncrtik.  VA 

DDG52 

BARRr" 

Nortik.  VA 

FFG  36 

UNCERWOOD 

Mascot  FL 

DDG53 

JON  PAUL  JONES 

San  Dego.  CA 

FFG  35 

CURTS 

San  Dego.  CA 

DDG  SI 

CLHTTS  WLBLR 

Yokosuka.  Jaoan 

FFG  39 

DCYLE 

Mascot  FL 

DDG55 

STOUT 

Nortik.  VA 

FFG  40 

HALY8URTON 

Mascot  FL 

DDG56 

JON  S  MCCAIN 

Yckosuka.  Japan 

FFG  41 

MCOJJSKY 

San  Dego.  CA 

DDG57 

MTTSOER 

Nortik.  VA 

FFG  42 

KLAKRING 

Mascot  R. 

DDG5S 

LABOON 

Nortik.  VA 

FFG  43 

THACH 

San  Dego.  CA 

DDGS9 

RUSSELL 

Pear  Harbo.  HI 

FFG  45 

DG  WERT 

Mascot  FL 

DDG  50 

PAUL  HAMLTON 

Peaif  Harbo.  HI 

FFG  46 

RENTZ 

San  Dego.  CA 

OCXS  61 

RAMAGE 

Nortik.  VA 

FFG  47 

NICHOLAS 

ICrfcik.  VA 

DDG53 

STETHEM 

Yokosuka. Jaoan 

FFG  45 

VANCEGRJFT 

San  Dego.  CA 

(V)7 

SH*> 

HOMEPORT 

DDG  64 

CARfEY 

Mascot  FL 

FFG  49 

ROBERT  G BRADLEY 

Mascot  FL 

CG  66 

HUE  QTY 

Mascot  FL 

DDG55 

BENFDLD 

San  Dego.  CA 

FFG  53 

HAVES 

fertile.  VA 

CG  67 

SHUDH 

YokosJca.  Japan 

DDG  56 

G0N3LLEZ 

Nortik.  VA 

FFG  55 

ELROD 

fertik.  VA 

DDG  67 

COLE 

Ncrtik.  VA 

FFG  56 

SMPSCN 

Mascot  FL 

(V)9 

SHi> 

HOMEPORT 

DDG  55 

T1-E  SULLIVANS 

San  Dego.  CA 

FFG  57 

REUBEN  JAMES 

F^ai  Harbo.  HI 

FFG  37 

CRCMMEUN 

Peat!  Harbor.  HI 

DDG  55 

ULUS 

San  Dego.  CA 

FFG  55 

SAMUEL  B  ROBERTS 

Mascot  FL 

FFG  50 

TAVLOR 

Mascot  FL 

DDG  70 

HOPPER 

Peak  Harbo.  HI 

FFG  59 

KAUFFMAN 

fertile  VA 

FFG  51 

GARY 

San  Dego.  CA 

DDG  71 

ROSS 

Ncrtik.  VA 

FFG  60 

ROONEY  M.  DAVIS 

Everett.  WA 

FFG  52 

CARR 

Nortik.  VA 

DDG  72 

MAHAN 

Nortik.  VA 

FFG  61 

INGRAHAM 

Everett.  WA 

FFG  54 

FORD 

Eerea.  WA 

DDG  73 

DECATUR 

San  Dego.  CA 

DDG  74 

MCFAUL 

Nortik.  VA 

DDG  75 

DONALD  COOK 

Nortik.  VA 

DDG  76 

HIGGINS 

San  Dego.  CA 

DDG  77 

OKAfE 

Peat  Harbo.  HI 

DDG  75 

PORTER 

Nortik.  VA 

All  sonar  technicians-surface  (STGs)  attend  STG  A  school.  At  A  school,  students 
learn  the  basic  principles  of  the  STG  rating  including  oceanography  and  principles  of  sound. 
Following  A  school,  STGs  are  sent  to  different  courses  depending  on  whether  they  are 
operators  or  operator/maintainers.  STGs  who  are  strictly  operators  are  sent  to  a  sonar 
operator  course,  where  they  learn  how  to  operate  the  specific  89  variant  of  the  ship  to  which 
they  will  be  sent.  Maintainers  are  sent  to  C  school,  where  they  learn  the  technical  skills 
required  to  maintain  the  equipment  they  will  work  on  upon  reporting  to  their  ship  (Navy 
Personnel  Command,  2012). 

CBT  was  introduced  full-time  into  the  training  pipeline  in  FY2003,  after  the 
recommendations  of  the  ERNT  report  (Naval  Inspector  General,  2009).  Data  were  not 
available  to  show  how  STG  course  lengths  were  affected  by  the  conversion  to  CBT.  The 
2009  Navy  IG  report  examined  the  course  lengths  of  22  A  and  C  schools  for  ILT  and  CBT 
and  found  that  on  average,  CBT  course  lengths  were  26%  shorter  than  ILT  course  lengths 
(Naval  Inspector  General,  2009).  This  study  focuses  on  FY1999  through  FY2010  to  capture 
data  prior  to  and  after  the  introduction  of  CBT.  Initially,  FY1995  through  FY1998  were  also 
considered,  but  there  were  not  enough  data  available  during  this  time  frame  for  most  data 
categories.  The  raw  data  provided  were  analyzed  to  reveal  relationships  between  selected 
data  sets. 

Program  Executive  Office  Integrated  Warfare  System  5  (PEO  IWS5)  provided  a  list 
of  ships  equipped  with  the  AN/SQQ-89(v)  sonar  system.  The  list  included  ship  class,  ship 
name,  hull  number,  homeport,  and  89  variant  number.  Only  ships  with  AN/SQQ-89(v) 
variants  on  board  both  before  and  after  implementation  of  CBT  were  considered.  The  initial 
list  provided  by  PEO  IWS5  included  all  ships  of  the  CG-47,  DD-963,  DDG-51,  and  FFG-7 
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classes.  To  narrow  the  ship  list  to  match  the  scope  of  our  study,  ships  were  removed  from 
the  data  set  if 

•  the  ship  was  decommissioned  during  the  FY1995-FY2006  time  frame, 

•  the  ship  received  a  variant  upgrade, 

•  the  ship  was  commissioned  FY2000  or  later,  or 

•  the  ship  was  outfitted  with  a  variant  introduced  after  FY2003. 

Using  these  criteria,  the  ship  list  was  reduced  to  68  ships.  VAMOSC  provided  O&S  cost 
data,  underway  steaming  days,  and  selected  non-cost  data  for  ships  equipped  with  the 
AN/SQQ-89  sonar  system  covering  FY1995  through  FY2010.  Cost  figures  were  given  in 
then-year  and  constant  FY201 1  dollars. 

In  addition  to  the  overall  89  system  data,  detailed  ship  data  were  available  for  the 
selected  ships.  Non-cost  data  included  number  of  personnel  trained,  maintenance 
manhours,  and  number  of  maintenance  actions.  The  data  were  used  to  calculate  average 
time  (in  manhours)  spent  per  maintenance  action.  These  numbers  were  calculated  to 
determine  training  and  maintenance  trends  pre-  and  post-CBT  (see  Figure  1).  For  instance, 
if  the  average  time  spent  per  maintenance  action  increased,  it  could  suggest  a  backlog  of 
maintenance  or  a  lack  of  technical  competence  in  performing  a  maintenance  action.  Prior  to 
CBT,  manhours  spent  per  maintenance  action  were  trending  upward;  after  the  introduction 
of  CBT,  manhours  per  maintenance  action  remained  relatively  flat.  This  suggests  that 
manhours  per  maintenance  action  may  have  reacted  positively  to  the  conversion  to  CBT; 
however,  this  assumes  that  the  types  of  maintenance  actions  performed  remained  relatively 
constant. 


Single-factor  regression  analysis  was  performed  to  explore  the  relationship  between 
total  training  cost  and  the  O&S  variables  provided  by  Navy  VAMOSC.  Unit  Level 
Consumption,  IM,  Equipment  Rework,  and  Depot  Maintenance  were  selected  as  variables 
to  determine  whether  changes  in  training  costs  resulted  in  increased  maintenance  costs. 
The  selected  variables  represent  organizational-level,  intermediate,  and  depot-level 
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maintenance.  Unit  Level  Consumption  is  a  summation  of  Organizational  Repair  Parts, 
Replenishment  Spares,  and  Logistics  Center  (LOGCEN)  exchanges.  Equipment  Rework  is  a 
summation  of  contractor  and  government  Program  Office  rework  costs.  IM  is  a  summation  of 
afloat  and  ashore  IM  labor  costs  and  ashore  IM  materials  costs.  Depot  Maintenance  is  a 
summation  of  private  and  public  shipyard  depot  costs.  Training  (Total)  is  a  summation  of 
Program  Office  and  NETPDTC  training  costs.  The  results  are  shown  in  Table  2. 

Table  2.  Regression  Analysis — O&S  Components 


R^ 

Unit  Level 

Consumption 

0.033 

Equipment  Rework 

0.002 

Intermediate 

Maintenance 

0.348 

Depot-Level 

Maintenance 

0.208 

In  this  case,  regression  analysis  shows  that  none  of  the  factors  selected  have  a 
strong  relationship  to  total  training  cost,  suggesting  that  if  maintenance  costs  are  related  to 
training  costs  in  the  STG  rating,  there  are  other  factors  not  identified  in  this  study  that  are 
having  an  effect.  It  is  interesting  to  look  at  the  relationship  of  IM  costs  and  training  dollars  in 
a  scatter  diagram  (Figure  2). 


Training  vs  Intermediate  Maintenance 
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Figure  2.  Training  vs.  Intermediate  Maintenance  Scatter  Plot 


Figure  2  suggests  that  there  may  be  a  weak  relationship  between  training  dollars  and 
intermediate  maintenance  costs  and  that  this  warrants  further  investigation.  Specifically,  the 
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plot  suggests  that  as  training  costs  increase,  intermediate  maintenance  costs  decrease. 
This  would  support  the  hypothesis  that  when  less  money  is  spent  on  training  (as  a  result  of 
switching  to  CBT),  the  maintenance  costs  will  increase. 


Graphical  analysis  of  several  data  categories  indicated  noticeable  changes  after  the 
introduction  of  CBT.  For  example,  Labor  Ashore — Intermediate  Maintenance  Manhours 
showed  significant  change  (see  Figure  3). 


Figure  3.  Labor  Ashore — Intermediate  Maintenance  Manhours 


The  figure  shows  the  number  of  manhours  spent  on  IM  for  selected  ships  from 
FY1995  through  FY2010.  Beginning  in  FY2004,  the  IM  manhours  increased  significantly  for 
the  selected  DDG-51  and  CG-47  class  ships.  While  this  may  be  partially  explained  by 
changes  to  Navy  maintenance  policy  described  earlier,  it  corroborates  the  evidence 
suggested  in  Figure  2,  namely  that  as  CBT  is  introduced  and  training  costs  decrease, 
intermediate  maintenance  hours  and  cost  increase.  This  trend  was  not  as  evident,  however, 
for  the  FFG-7  class.  This  may  be  explained  by  funding  of  the  ship  class,  since  many  of  these 
ships  belong  to  the  Naval  Reserve  Force  and  it  is  likely  that  their  funding  levels  did  not 
change  throughout  the  period  studied. 

Paired  t-tests  were  conducted  to  determine  whether  the  means  of  paired 
observations  (selected  ships  pre-  and  post-CBT)  were  different.  The  null  hypothesis  is  that 
there  is  no  significant  difference  in  the  means  before  and  after  the  introduction  of  CBT.  The 
alternate  hypothesis  is  that  there  is  a  significant  difference  between  the  means  due  to  CBT. 
A  negative  t-statistic  indicates  that  the  pre-CBT  mean  was  smaller,  while  a  positive  t-statistic 
indicates  that  the  post-CBT  mean  was  smaller.  Response  variables  used  in  this  study  were 
corrective  organizational  and  IM  actions,  organizational  parts  cost,  exchanges  LOGCEN 
cost,  manhours  organizational  labor,  and  labor  ashore  IM  manhours  (see  Table  3). 
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Table  3.  Paired  t-Test  Results 


Variable 

degrees 

freedom 

Number  of 
obs. 

Mean 

t  statistic 

p-value 

Corrective  org.  & 
IM  actions 

786 

before  CBT 

335 

61.43 

-6.61 

0.0000 

after  CBT 

458 

85.10 

Organizational 
parts  cost 

790 

before  CBT 

336 

6914.4 

-4.07 

0.0000 

after  CBT 

466 

9539.6 

Exchanges 
LOGCEN  cost 

563 

before  CBT 

238 

43752 

-6.30 

0.0000 

after  CBT 

328 

72178 

Manhours  org. 
labor 

779 

before  CBT 

335 

739.9 

-7.03 

0.0000 

after  CBT 

466 

1208.7 

Labor  ashore  IM 
manhours 

206 

before  CBT 

109 

107.51 

-5.17 

0.000 

after  CBT 

149 

357.17 

In  all  cases,  the  p-values  were  less  than  0.01,  indicating  a  significant  difference 
between  the  means  pre-  and  post-CBT.  This  suggests  that  the  introduction  of  CBT  had  a 
statistically  significant  impact  on  several  measures  of  maintenance  activity  and  cost. 


Interestingly,  the  number  of  maintenance  actions  (organizational  and  IM)  increased, 
even  though  changes  in  Navy  maintenance  policies  would  have  initially  led  to  fewer 
maintenance  actions.  Since  we  report  the  average  over  several  years,  it  is  possible  that  the 
expected  increase  in  future  maintenance  actions  was  part  of  the  observed  mean  after  CBT. 
It  is  also  likely  that  changes  in  operating  tempo  (due  to  the  GWOT)  had  a  significant  impact 
on  this  variable  as  well,  so  it  is  not  possible  to  isolate  the  effect  of  CBT  on  corrective 
maintenance  actions. 

Most  interesting  are  the  three  categories  related  to  maintenance  actions  performed 
by  sailors  at  the  ship  level:  organizational  parts  cost,  exchanges  LOGCEN  cost,  and 
manhours  organizational  labor.  If  the  conversion  to  CBT  were  to  have  an  effect  anywhere  in 
the  Navy  maintenance  system,  it  would  be  at  maintenance  activities  where  sailors  were 
performing  maintenance  on  ships.  This  data  would  support  the  anecdotal  evidence  provided 
by  ship  operators  that  CBT  training  would  also  impact  labor  ashore  IM  manhours  (in  IM 
facilities);  however,  a  confounding  variable  for  IM  is  that  several  shore-based  IM  facilities 
were  closed  and  shore-based  IM  billets  for  sailors  were  eliminated. 

Conclusion  and  Areas  for  Further  Research 

In  2001,  ERNT  released  its  report,  Revolution  in  Training:  Executive  Review  of  Navy 
Training  Final  Report,  which  led  to  a  major  overhaul  in  the  U.S.  Navy’s  training  practices, 
including  the  use  of  CBT  in  A  and  C  schools.  While  government  studies  of  the  Navy’s  CBT 
training  confirmed  that  the  transition  to  CBT  resulted  in  shorter  training  times  and  cost 
savings,  sailors  reporting  to  the  fleet  were  not  as  well  prepared  as  classroom-trained  sailors 
of  the  past,  and  extensive  OJT,  supervision,  and  assistance  in  performing  basic 
maintenance  tasks  were  required  to  bring  CBT-trained  sailors  up  to  speed. 

During  the  same  period  of  time  that  CBT  was  being  implemented,  the  U.S.  Navy 
reorganized  its  maintenance  program  and  reduced  total  manning  levels  on  ships,  even 
though  ship  requirements  did  not  change,  meaning  that  ships  had  to  do  more  work  with 
fewer  personnel.  Many  maintenance  requirements  were  deferred  or  eliminated  on  ships  with 
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the  expectation  that  shore  facilities  would  pick  up  the  slack,  but  shore  facilities  also 
experienced  manning  reductions.  As  a  result,  less  planned  maintenance  was  being 
performed  on  equipment,  which  increased  opportunities  for  equipment  failure  and 
decreased  fleet  material  readiness. 

This  study  looked  at  costs  from  a  systems  perspective,  considering  not  only  the  cost 
of  training  but  also  the  cost  of  maintenance.  We  asked  the  following  question:  If  sailors 
trained  with  CBT  had  lower  knowledge  and  skill  levels,  did  this  contribute  to  increased 
operations  and  maintenance  costs? 

Unfortunately,  there  were  too  many  confounding  variables  that  could  have  affected 
operation  and  maintenance  costs  during  this  period  of  time  to  draw  any  conclusions  about 
the  effect  of  CBT  on  maintenance  costs  from  the  Navy  level.  Instead,  we  focused  on  a 
single  Navy  system,  the  AN/SQQ-89(v)  sonar  system,  to  examine  the  effects  of  the 
conversion  to  CBT  on  maintenance. 

The  results  of  the  study  revealed  several  pieces  of  useful  information.  Regression 
analysis  indicates  a  weak  relationship  between  decreasing  in  training  costs  and  an 
increasing  in  IM  costs.  In  addition,  paired  t-tests  showed  that  the  conversion  to  CBT  may 
have  led  to  increases  in  corrective  organizational  and  IM  actions,  organizational  parts  cost, 
exchanges  LOGCEN  cost,  manhours  organizational  labor,  and  labor  ashore  IM  manhours. 
Of  particular  interest  were  results  for  manhours  organizational  labor,  organizational  parts 
cost,  and  exchanges  LOGCEN  cost,  all  associated  with  maintenance  performed  by  sailors 
at  the  unit  (ship)  level,  because  conversion  to  CBT  training  would  be  most  noticeable  at 
maintenance  activities  where  sailors  are  performing  the  maintenance. 

As  the  Navy  IG,  GAO,  and  Balisle  reports  suggest,  there  are  several  factors  that 
have  contributed  to  declines  in  fleet  readiness.  Most  notably,  the  simultaneous  combination 
of  changes  in  training,  maintenance,  and  manning  policies  appear  to  have  had  lasting 
negative  impacts,  including  rising  fleet  maintenance  costs.  The  data  analysis  performed  in 
this  study  shows  that  the  change  to  CBT  was  statistically  significant  when  compared  to 
several  maintenance  variables,  but  it  is  also  likely  that  changes  to  all  three  areas  (training, 
maintenance,  and  manning)  had  collective  negative  effects  which  go  much  further  than 
rising  maintenance  costs  and  actions.  It  is  clear  that  policy  changes  in  the  2000s  impacted 
fleet  readiness  in  a  negative  manner,  but  no  clear  conclusions  can  be  drawn  about  the 
specific  impact  of  CBT  on  total  system  cost  from  the  data  examined  in  this  study. 

Because  the  data  collected  can  be  characterized  as  panel  data,  statistical  analysis 
that  recognizes  the  panel  nature  of  the  data  will  be  performed  and  reported  in  another 
paper.  It  may  be  useful  to  study  the  impacts  of  the  conversion  to  CBT  on  other  Navy 
systems.  From  a  training  perspective,  the  lack  of  measures  of  effectiveness  for  training  may 
prove  frustrating  in  drawing  any  conclusions,  but  from  a  cost  perspective,  it  may  be  possible 
to  gain  further  insight  into  the  types  of  cost  most  affected  by  CBT. 
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The  GAO’s  11th  Annual  Assessment  of  Selected  Weapon 

Programs 


Michael  J.  Sullivan — Sullivan  is  the  director,  Acquisition  and  Sourcing  Management,  U.S. 
Government  Accountability  Office.  This  group  has  responsibility  for  examining  the  effectiveness  of  the 
DoD’s  acquisition  and  procurement  practices  in  meeting  its  mission  performance  objectives  and 
requirements.  In  addition  to  directing  reviews  of  major  weapon  system  acquisitions  such  as  the  Joint 
Strike  Fighter,  F-22,  Global  Hawk,  and  various  other  major  weapon  acquisition  programs,  Sullivan 
has  developed  and  directs  a  body  of  work  examining  how  the  DoD  can  apply  best  practices  to  the 
nation’s  largest  and  most  technically  advanced  weapon  systems  acquisition  system.  This  work  has 
spanned  a  broad  range  of  issues  critical  to  the  successful  delivery  of  systems,  including  technology 
development,  product  development,  transition  to  production,  software  development,  program 
management,  requirement-setting,  cost  estimating,  and  strategic  portfolio  management.  The  findings 
and  recommendations  from  this  work  have  played  a  major  role  in  the  department’s  recent  acquisition 
policy  revisions.  Most  recently,  he  has  directed  the  GAO’s  annual  assessment  of  major  weapon 
systems  programs  for  the  Congress  and  GAO’s  work  with  Congress  in  establishing  acquisition  policy 
reforms.  His  team  also  provides  the  Congress  with  early  warning  on  technical  and  management 
challenges  facing  these  investments.  Sullivan  has  been  with  the  GAO  for  24  years.  He  received  a 
bachelor's  degree  in  political  science  from  Indiana  University  and  a  master’s  degree  in  public 
administration  from  the  School  of  Public  and  Environmental  Affairs,  Indiana  University. 
[sullivanm@gao.gov] 

Abstract 

This  paper  reflects  the  GAO’s  observations  on  how  well  the  DoD  is  planning  and  executing  its 
$1 .602  trillion  portfolio  of  major  weapon  programs.  Although  the  total  projected  cost  of  the 
portfolio  remains  significant,  that  cost  has  declined  since  peaking  at  $1.75  trillion  in  2010  and 
is  currently  at  its  lowest  point  in  over  five  years.  In  addition,  the  number  of  programs  in  the 
portfolio  has  decreased  from  98  programs  in  2010  to  86  programs  in  2012.  DoD  weapon 
system  acquisition  represents  one  of  the  largest  areas  of  the  government’s  discretionary 
spending.  With  the  likelihood  of  decreased  defense  budgets  looming  in  the  near  future,  it  is 
imperative  that  the  DoD  continue  to  find  ways  to  reduce  cost  and  improve  efficiency. 

Introduction 

This  paper  reflects  the  GAO’s  observations  on  how  well  the  DoD  is  planning  and 
executing  its  $1,602  trillion  portfolio  of  major  weapon  programs.  Although  the  total  projected 
cost  of  the  portfolio  remains  significant,  that  cost  has  declined  since  peaking  at  $1.75  trillion 
in  2010  and  is  currently  at  its  lowest  point  in  over  five  years.  In  addition,  the  number  of 
programs  in  the  portfolio  has  decreased  from  98  programs  in  2010  to  86  programs  in  2012. 
DoD  weapon  system  acquisition — an  area  that  has  been  on  the  GAO’s  high-risk  list  for  more 
than  20  years — still  represents  one  of  the  largest  areas  of  the  government’s  discretionary 
spending.  Over  the  past  decade,  Congress  and  the  DoD  have  made  meaningful 
improvements  in  the  statutory  and  policy  frameworks  that  govern  weapon  system 
acquisitions  by  mandating  and  encouraging  a  more  knowledge-based  approach  to  the 
development  and  production  of  major  systems.  The  GAO  has  noted  in  the  past  that  practice 
has  lagged  behind  policy  in  certain  areas,  and  commensurate  improvements  in  program 
outcomes  have  not  been  evident.  However,  the  changes  in  the  DoD’s  portfolio  over  the  past 
few  years  indicate  that  some  improvements  are  being  realized.  With  the  likelihood  of 
decreased  defense  budgets  looming  in  the  near  future,  it  is  imperative  that  the  DoD  continue 
to  find  ways  to  reduce  cost  and  improve  efficiency. 

The  following  are  observations  on  (1)  the  cost  and  schedule  performance  of  the 
DoD’s  2012  portfolio  of  86  major  defense  acquisition  programs,  including  the  Missile 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  358- 


Defense  Agency’s  (MDA’s)  Ballistic  Missile  Defense  System  (BMDS);  (2)  the  knowledge 
attained  at  key  junctures  in  the  acquisition  process  for  40  weapon  programs  in  development 
or  early  production;  and  (3)  key  acquisition  reform  initiatives  and  program  concurrency.1  For 
a  more  detailed  discussion  of  each  of  the  observations,  see  GAO-1 3-294SP  (GAO,  2013). 

Data  from  three  sets  of  programs  provided  the  basis  for  the  observations: 

•  We  assessed  all  86  major  defense  acquisition  programs  in  the  DoD’s  2012 
portfolio  for  our  analysis  of  cost  and  schedule  performance.  To  develop  our 
observations,  we  obtained  cost,  schedule,  and  quantity  data  from  the  DoD’s 
December  2011  Selected  Acquisition  Reports  (SARs)  and  from  the  Defense 
Acquisition  Management  Information  Retrieval  Purview  system.  In  order  to 
fully  reflect  the  total  size  and  cost  of  the  DoD’s  portfolio,  we  included  the  cost 
of  BMDS — as  of  DoD’s  fiscal  year  2013  budget  submission — in  this  year’s 
assessment  of  the  changes  in  the  overall  cost  and  size  of  the  portfolio  over 
the  past  year.  However,  the  program  was  excluded  from  the  remainder  of  our 
analyses  because  no  acquisition  program  baseline  exists. 

•  We  assessed  40  MDAPs  that  were  mostly  between  the  start  of  development 
and  full-rate  production  for  our  analysis  of  knowledge  attained  at  key 
junctures  and  the  implementation  of  acquisition  reforms.  To  develop  our 
observations,  we  obtained  information  on  the  extent  to  which  the  programs 
follow  knowledge-based  practices  for  technology  maturity,  design  stability, 
and  production  maturity  using  a  data-collection  instrument.  We  also 
submitted  a  survey  to  program  offices  to  collect  information  on  systems 
engineering  reviews,  design  stability,  manufacturing  planning  and  execution, 
and  the  implementation  of  specific  acquisition  reforms.  We  received  survey 
responses  from  all  of  the  programs  from  August  to  November  201 2. 

•  In  addition,  we  assessed  17  future  major  defense  acquisition  programs  in 
order  to  gain  additional  insights  into  the  implementation  of  key  acquisition 
reform  initiatives.  To  develop  our  observations,  we  submitted  a  survey  to 
program  offices  to  collect  information  on  program  schedule  events,  costs,  and 
numerous  acquisition  reforms,  and  received  responses  from  all  17  future 
programs  from  August  to  October  2012. 

Observations  on  the  Performance  of  the  DoD’s  2012  Major  Defense  Acquisition 
Program  Portfolio 

The  overall  size  and  estimated  cost  of  the  DoD’s  portfolio  of  MDAPs  decreased  over 
the  past  year,  while  the  average  time  to  deliver  initial  capability  to  the  warfighter  increased 
by  one  month.  Our  analysis  of  the  DoD’s  2012  portfolio  allows  us  to  make  the  following  nine 
observations. 

1 .  The  DoD’s  2012  portfolio  of  major  defense  acquisition  programs  contains  86 
programs  with  a  combined  total  estimated  cost  of  $1 .602  trillion,  which  is  a 


1  Major  defense  acquisition  programs  are  those  identified  by  the  DoD  that  require  eventual  total 
research,  development,  test,  and  evaluation  (RDT&E)  expenditures,  including  all  planned  increments, 
of  more  than  $365  million,  or  procurement  expenditures,  including  all  planned  increments,  of  more 
than  $2.19  billion,  in  fiscal  year  2000  constant  dollars.  The  DoD  has  a  list  of  programs  designated  as 
pre-major  defense  acquisition  programs  (pre-MDAP).  These  programs  have  not  formally  been 
designated  as  MDAPs;  however,  the  DoD  plans  for  these  programs  to  enter  system  development,  or 
bypass  development  and  begin  production,  at  which  point  they  will  likely  be  designated  as  MDAPs. 
We  refer  to  these  programs  as  future  major  defense  acquisition  programs  throughout  this  report. 
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reduction  of  10  programs  and  more  than  $152  billion  from  201 1  levels.  This 
represents  the  smallest  portfolio  in  more  than  five  years.2 

2.  The  total  estimated  acquisition  cost  of  the  86  programs  in  the  201 2  portfolio 
decreased  by  $44  billion  over  the  past  year  while  the  delivery  of  initial 
operating  capability  slipped  by  one  month  on  average.3  When  assessed 
against  first  full  estimates,  the  total  cost  of  the  portfolio  has  increased  by  over 
$400  billion,  including  more  than  $90  billion  in  development  cost  growth  and 
nearly  $290  billion  in  procurement  cost  growth,  with  an  average  delay  of  27 
months  in  the  delivery  of  initial  operating  capability.4 

3.  Program  cancelations  and  restructurings  account  for  nearly  all  of  the  cost 
reduction  over  the  past  year. 

4.  Long-term  progress  of  the  Missile  Defense  Agency’s  $133  billion  BMDS 
cannot  be  assessed  because  insight  into  future  program  costs  is  limited  to 
the  five  years  covered  by  the  budget,  and  the  program  was  not  required  to 
establish  an  acquisition  program  baseline  when  it  began. 

5.  More  than  60%  of  the  programs  in  the  2012  portfolio  increased  buying  power 
over  the  past  year — as  measured  by  a  decrease  in  program  acquisition  unit 
cost — a  notable  improvement  when  compared  to  last  year,  when  more  than 
60%  of  the  programs  in  the  portfolio  lost  buying  power. 

6.  When  measured  against  cost  growth  targets  discussed  by  the  DoD,  the 
Office  of  Management  and  Budget  (OMB),  and  the  GAO,  the  portfolio’s 
performance  has  improved.  Only  15%  of  programs  exceeded  the  one-year 
target — down  from  40%  last  year — and  smaller  percentages  of  programs 
exceeded  targets  for  growth  both  in  the  past  five  years  and  since  first  full 
estimates. 

7.  Eight  of  the  10  costliest  programs  in  the  DoD’s  portfolio,  excluding  BMDS, 
reported  cost  reductions  over  the  past  year  totaling  nearly  $5  billion — about 
10%  of  the  portfolio’s  total  cost  reduction. 

8.  The  DoD  has  invested  more  than  $805  billion  in  its  2012  portfolio,  leaving 
over  $660  billion  remaining  to  be  funded,  excluding  BMDS.  More  than  90%  of 
the  remaining  funding  is  for  procurement,  with  more  than  60%  of  that  amount 
associated  with  the  10  costliest  programs  in  the  portfolio,  most  prominently 
the  Joint  Strike  Fighter. 

9.  Around  40%  of  the  funding  needed  to  complete  the  programs  in  the  portfolio 
represents  cost  growth  since  first  full  estimates. 

Observations  on  Knowledge  Attained  by  Programs  at  Key  Acquisition  Junctures 

Our  2013  assessment  continues  to  demonstrate  both  progress  and  a  significant 
need  for  programs  to  better  follow  a  knowledge-based  approach,  reducing  gaps  in 
technology,  design,  and  production  knowledge.  Knowledge  in  these  three  areas  builds  over 


2  All  dollar  figures  are  in  fiscal  year  2013  constant  dollars,  unless  otherwise  noted. 

3  In  addition  to  research  and  development  and  procurement  costs,  total  acquisition  cost  includes 
acquisition-related  operation  and  maintenance  and  system-specific  military  construction  costs. 

4  Our  discussion  of  cost  growth  since  first  full  estimates  does  not  include  the  BMDS,  as  the  program 
was  not  required  to  establish  an  acquisition  program  baseline  when  it  began  (see  GAO,  2012a,  for  an 
assessment  of  the  Missile  Defense  Agency’s  cost,  testing,  and  performance  progress  in  developing 
the  system). 
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time — a  knowledge  deficit  early  in  a  program  can  cascade  through  design  and  production, 
leaving  decision-makers  with  less  knowledge  to  support  decisions  about  when  and  how  best 
to  move  into  subsequent  acquisition  phases  that  commit  more  budgetary  resources.  A 
knowledge-based  acquisition  approach  is  a  cumulative  process  in  which  certain  knowledge 
is  acquired  by  key  decision  points  before  proceeding.  Demonstrating  technology  maturity  is 
a  prerequisite  for  moving  forward  into  system  development,  during  which  the  focus  should 
be  on  design  and  integration.  A  stable  and  mature  design  is  likewise  a  prerequisite  for 
moving  forward  into  production  where  the  focus  should  be  on  efficient  manufacturing.  We 
assessed  the  knowledge  attained  at  key  junctures  in  the  acquisition  process  for  40  individual 
weapon  programs,  which  are  mostly  in  development  or  early  production.5  Not  all  programs 
included  in  our  assessment  of  knowledge-based  practices  provided  information  for  every 
knowledge  point  or  had  reached  all  of  the  knowledge  points — development  start,  design 
review,  and  production  start — at  the  time  of  our  review.  Our  analysis  allows  us  to  make  the 
following  three  observations: 

1 .  Many  of  the  programs  that  began  in  the  last  five  years  had  mature 
technologies  and  held  a  preliminary  design  review  prior  to  the  start  of 
development  (knowledge  point  1),  providing  a  better  foundation  to  avoid 
future  cost  and  schedule  problems. 

2.  Less  than  one  third  of  the  programs  that  provided  data  on  design  drawings 
released  actually  reported  having  a  stable  design  at  their  critical  design 
review  (knowledge  point  2),  and  the  use  of  other  knowledge-based  practices 
to  ensure  design  stability  at  this  critical  juncture  varied. 

3.  Many  of  the  programs  we  assessed  have  taken  or  plan  to  take  steps  to 
capture  critical  manufacturing  knowledge  prior  to  the  start  of  production 
(knowledge  point  3),  although  the  methods  used  varied. 

Observations  on  Implementation  of  Acquisition  Reform  Initiatives  and  Program 
Concurrency 

Over  the  past  several  years,  Congress  and  the  DoD  have  instituted  multiple 
initiatives  aimed  at  improving  the  way  the  department  does  business  by  driving  down 
acquisition  costs  and  ensuring  that  programs  are  more  affordable:  specifically  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009,  the  reissuance  of  DoD  Instruction  5000.02 
(OUSD[AT&Lj,  2008),  and  multiple  “Better  Buying  Power”  memorandums  issued  by  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (OUSD[AT&L], 

2010a,  2010b,  2010c,  2012).  We  analyzed  survey  data  collected  from  40  current  major 
defense  acquisition  programs — the  same  programs  reflected  in  our  knowledge  point 
analysis — and  17  programs  identified  by  the  DoD  as  future  major  defense  acquisition 
programs,  regarding  the  implementation  of  key  aspects  of  these  reform  initiatives.  We 
focused  our  analysis  on  the  aspects  of  the  DoD’s  “Better  Buying  Power”  initiatives  and  the 
Weapon  Systems  Acquisition  Reform  Act  of  2009  aimed  at  ensuring  program  and  portfolio 
affordability,  controlling  cost  growth,  and  promoting  competition  throughout  the  acquisition 
life-cycle.6  In  addition,  we  assessed  the  amount  of  concurrency  between  developmental 


5  Because  knowledge  points  and  best  practices  differ  for  shipbuilding  programs,  we  exclude  the  six 
shipbuilding  programs  from  some  of  our  analysis  related  to  knowledge  point  2  and  knowledge  point  3. 

6  In  December  2012,  we  reported  on  the  DoD’s  implementation  of  the  Weapon  System  Acquisition 
Reform  Act  of  2009  and  noted  that  the  DoD  had  taken  steps  to  implement  fundamental  provisions  of 
the  Act,  and  that  the  DoD  was  taking  additional  steps  to  further  strengthen  its  policies  and  acquisition 
capabilities.  We  also  reported,  however,  that  the  DoD  still  faced  organizational,  policy,  and  cultural 
challenges  to  implementing  acquisition  reform  (GAO,  2012b). 
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testing  and  production  for  those  current  programs  beyond  knowledge  point  3. 7  We  have 
consistently  emphasized  the  importance  of  completing  developmental  testing  before 
entering  production  and  have  pointed  out  the  increased  risks  associated  with  concurrent 
testing  and  production.  Our  analysis  allows  us  to  make  the  following  five  observations: 

1 .  The  implementation  of  several  key  initiatives  in  the  Weapon  System 
Acquisition  Reform  Act  of  2009  aimed  at  increasing  program  knowledge  at 
development  start  varied  among  the  future  major  defense  acquisition 
programs  we  surveyed. 

2.  Around  half  of  the  current  and  future  programs  we  assessed  have 
established  affordability  requirements,  and  many  are  meeting  those 
requirements. 

3.  Almost  90%  of  the  current  MDAPs  we  assessed  have  conducted  “should 
cost”  analysis,  and  most  of  those  programs  noted  that  they  had  realized  or 
expected  to  realize  cost  savings  as  a  result. 

4.  Although  the  DoD  recognizes  the  need  for  and  benefits  of  competition  in 
weapon  system  acquisitions,  and  the  Weapon  Systems  Acquisition  Reform 
Act  of  2009  requires  programs  to  have  competitive  acquisition  strategies, 
many  of  the  programs  we  assessed  did  not  have  such  strategies  in  place. 

5.  Nearly  80%  of  the  programs  we  surveyed  that  were  in  production  reported 
that  30%  or  more  of  their  developmental  testing  had  been  or  was  going  to  be 
done  during  production,  despite  the  increased  risk  that  design  changes  and 
costly  retrofits  will  need  to  be  made. 
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Abstract 

DoD  spending  on  services  has  been  trending  upwards  for  over  a  decade  and,  as  of  201 1 ,  it 
accounted  for  56%  of  total  contract  spending.  The  increased  reliance  on  services  contractors 
has  prompted  the  GAO  to  look  more  closely  at  the  acquisition  and  contract  management 
process.  In  this  research,  we  address  the  following  questions:  (1)  How  do  different 
stakeholders  define  successful  services  contracts  within  the  Navy?  (2)  How  do  different 
stakeholders  measure  services  contracts  within  the  Navy?  and  (3)  How  should  Navy  services 
contracts  be  defined  and  measured?  We  conducted  a  survey  of  168  key  stakeholders.  We 
discovered  that  when  defining  and  measuring  the  success  of  a  service  contract,  all 
stakeholders  tend  to  utilize  outcome-related  factors  over  process-oriented  factors.  We  believe 
this  is  because  outcomes  tend  to  drive  perceptions  of  success  more  than  processes  and  are 
more  easily  quantifiable.  Metrics  used  to  measure  success  are  typically  related  to  cost, 
schedule,  and  performance.  Based  on  these  findings,  we  provide  recommendations  on 
establishing  better  internal  control  measures,  putting  in  place  an  operational  audit  process, 
and  creating  a  standardized  reporting  process. 

Introduction 

The  service  sector  represents  the  largest  and  the  fastest  growing  segment  of  the 
economies  of  the  U.S.  and  other  developed  countries.  This  growth  of  services  in  the  overall 
economy  is  also  mirrored  by  the  growth  of  services  acquisition  in  the  DoD.  For  example,  the 
DoD  obligations  on  contracts  have  more  than  doubled  between  fiscal  years  2001  and  2008 
to  over  $387  billion,  with  over  $200  billion  spent  just  for  services  in  2008  (GAO,  2009).  In 
conjunction  with  this  increase  in  defense  procurement  is  the  reduction  of  the  defense 
acquisition  workforce.  The  size  of  the  federal  workforce  decreased  from  2.25  million  in  1990 
to  1.78  million  in  2000  (GAO,  2002).  The  combination  of  the  increasing  defense 
procurement  workload  and  the  decreasing  size  of  the  government  workforce,  along  with  the 
complexities  of  an  arcane  and  convoluted  government  contracting  process,  have  created 
the  perfect  storm — an  environment  in  which  complying  with  government  contracting  policies 
and  adopting  contract  management  best  practices  has  not  always  been  feasible  (Rendon, 
2010).  Between  2001  and  2009,  the  GAO  issued  16  reports  related  to  trends,  challenges, 
and  deficiencies  in  defense  contracting.  During  this  same  time  frame,  the  DoD  Inspector 
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General  (DoDIG)  issued  142  reports  on  deficiencies  in  the  DoD  acquisition  and  contract 
administration  processes.  These  reports  have  identified  poor  contract  planning,  contract 
administration,  and  contractor  oversight  as  just  some  of  the  critically  deficient  areas  in  DoD 
contract  management.  Because  of  these  deficiencies,  the  GAO  has  identified  contract 
management  as  a  “high  risk”  area  for  the  federal  government  since  1990  and  continues  to 
identify  it  as  high  risk  (GAO,  2013). 

As  the  DoD’s  services  acquisition  continues  to  increase  in  scope  and  dollars,  the 
agency  must  give  greater  attention  to  proper  acquisition  planning,  adequate  requirements 
definition,  sufficient  price  evaluation,  and  proper  contractor  oversight  (GAO,  2002).  In  fact, 
as  stressed  in  a  recent  memorandum  for  acquisition  professionals  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L],  2010),  improving  the 
efficiency  of  the  acquisition  of  products  and  services  is  of  utmost  importance  to  the  DoD.  In 
some  ways,  the  issues  affecting  services  acquisition  are  similar  to  those  affecting  the 
acquisition  of  physical  supplies  and  weapon  systems.  However,  the  unique  characteristics  of 
services  and  the  increasing  importance  of  services  acquisition  offer  a  significant  opportunity 
for  conducting  research  in  the  management  of  services  acquisition  in  the  DoD. 

Research  Questions 

This  research  project  undertakes  a  focused,  in-depth  study  of  the  services 
acquisition  so  as  to  understand  how  success  of  service  acquisition  contracts  is  being 
defined  and  measured  in  the  Navy.  The  contract  management  process  is  performed  with 
inputs  from  the  different  functional  areas,  such  as  program  management,  contracting, 
financial,  logistics,  and  quality  assurance.  Each  of  these  project  team  members  represents 
different  stakeholders  and  are  therefore  likely  to  have  different  goals  and  objectives.  Hence, 
the  first  research  question  we  investigated  was  as  follows:  How  do  different  stakeholders 
define  successful  services  contracts  within  the  Navy?  To  develop  a  clear  understanding  of 
current  services  acquisition  practices,  we  also  investigated  the  second  research  question: 
How  do  different  stakeholders  measure  services  contracts  within  the  Navy?  Investigating  the 
previous  two  questions  helped  us  develop  recommendations  regarding  the  third  and  final 
research  question:  How  should  the  service  contract’s  success  be  measured?  The  next 
section  provides  a  literature  review  of  some  of  the  management  theories  informing  service 
supply  chain  management,  as  well  as  some  of  our  previous  research  on  DoD  services 
acquisition. 

Literature  Foundation 

The  academic  research  in  the  management  of  services  acquisition  is  founded  on 
several  economic  and  management  theories,  including  agency  theory  (Eisenhardt,  1989), 
transaction  cost  economics  (Williamson,  1979),  contractual  theory  (Luo,  2002),  service 
operations  and  supply  management  (Fitzsimmons  &  Fitzsimmons,  2006),  and  stakeholder 
theory  (Freeman,  1984;  Cleland,1986;  El-Gohary,  Osman,  &  El-Diraby,  2006).  We  refer  the 
reader  to  our  earlier  technical  report  (Apte  &  Rendon,  201 3)  for  a  survey  of  prior  academic 
research,  and  we  also  provide  a  summary  of  research  projects  carried  out  by  the  authors  in 
the  area  of  services  supply  chain. 

We  have  addressed  the  need  for  research  in  this  increasingly  important  area  of 
services  acquisition  by  undertaking  six  sponsored  research  projects  over  the  past  six  years. 
The  first  two  research  projects  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006;  Apte  &  Rendon,  2007) 
were  exploratory  in  nature,  aimed  at  understanding  the  types  of  services  being  acquired,  the 
associated  rates  of  growth  in  services  acquisition,  and  the  major  challenges  and 
opportunities  present  in  the  service  supply  chain. 
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The  next  two  research  projects  were  survey-based  empirical  studies  aimed  at 
developing  a  high-level  understanding  of  how  services  acquisition  is  currently  being 
managed  at  a  wide  range  of  Army,  Navy,  and  Air  Force  installations  (Apte,  Apte,  &  Rendon, 
2008;  Apte,  Apte,  &  Rendon,  2009).  The  analysis  of  survey  data  indicated  that  the  current 
state  of  services  acquisition  management  suffers  from  several  deficiencies,  including  deficit 
billet  and  manning  levels  (which  are  further  aggravated  by  insufficient  training  and  the 
inexperience  of  acquisition  personnel)  and  the  lack  of  strong  project-team  and  life-cycle 
approaches.  Our  research  (Apte,  Apte,  &  Rendon,  2010)  also  analyzed  and  compared  the 
results  of  the  primary  data  collected  in  two  previous  empirical  studies  involving  Army,  Navy, 
and  Air  Force  contracting  organizations  so  as  to  develop  a  more  thorough  and 
comprehensive  understanding  of  how  services  acquisition  is  being  managed  within 
individual  military  Services. 

As  a  result  of  these  research  projects  dealing  with  the  service  supply  chain  in  the 
DoD,  we  have  developed  a  comprehensive,  high-level  understanding  of  services  acquisition 
in  the  DoD,  have  identified  several  specific  deficiencies,  and  have  proposed  a  number  of 
concrete  recommendations  for  performance  improvement. 

Based  on  the  foundation  of  the  previously  mentioned  management  theories, 
conclusions  of  the  GAO  and  DoDIG  reports  (Seifert  &  Ermoshkin,  2010),  and  findings  of  our 
own  sponsored  research  projects  on  the  topic,  we  believe  that  the  success  of  service 
acquisition  contracts  is  significantly  influenced  by  four  broadly  defined  factors:  (1)  the  type 
and  quantity  of  services  being  outsourced  and  the  associated  amount  of  acquisition-related 
workload;  (2)  the  characteristics  of  contracts  being  awarded;  (3)  the  capacity  available  to 
carry  out  the  contracting,  project  management,  and  surveillance  work;  and  (4)  various 
management  practices,  such  as  use  of  project  team  or  life-cycle  approaches  and  so  forth.  A 
conceptual  model  indicating  the  interrelationship  among  these  factors  is  shown  in  Figure  1. 


Figure  1.  Drivers  of  Acquisition  Practices  and  Success  of  Service 

Contracts 
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As  shown  in  the  conceptual  diagram  of  Figure  1 ,  the  contract  characteristics  are 
affected  by  the  type  of  service  being  acquired,  while  the  management  practices  being  used 
are  influenced  by  the  services  being  acquired,  the  contract  characteristics,  and,  more 
importantly,  the  capacity  available  to  perform  the  acquisition  work.  The  success  of  services 
contracts,  in  turn,  is  affected  by  the  previously  mentioned  four  drivers.  Underlying  Figure  1  is 
the  fundamental  question  motivating  our  in-depth  research:  What  drives  the  success  of 
services  contracts?  This  fundamental  question  is,  of  course,  critically  important,  and  yet  it  is 
also  not  one  that  can  be  answered  easily  or  quickly.  We  believe  that,  generally,  in  the  case 
of  questions  related  to  complex  systems,  it  is  preferable  to  break  down  the  overall  system  in 
smaller  parts,  gain  an  understanding  of  the  functioning  of  each  part,  and  then  put  all  the 
pieces  together  to  better  understand  the  overall  system  and  answer  the  fundamental 
question.  That  is  what  we  plan  to  do  in  this  research  by  addressing  three  research 
questions:  (1)  understand  how  the  success  of  services  contracts  is  being  defined  by 
different  stakeholders,  (2)  identify  how  the  success  of  services  contracts  is  currently  being 
measured,  and  (3)  develop  specific  recommendations  on  how  the  success  of  services 
contracts  should  be  measured.  We  address  our  research  methodology  in  the  next  section. 

Research  Methodology 

With  the  assistance  of  our  MBA  thesis  students  (Hagan,  Spede,  &  Sutton,  2012),  we 
developed  and  deployed  a  data  collection  survey  instrument  to  collect  empirical  data  for 
answering  our  research  questions.  The  survey  was  deployed  to  the  various  stakeholders  at 
the  participating  commands.  We  then  analyzed  the  data  using  descriptive  statistics  to 
provide  recommendations  and  conclusions. 

We  developed  and  deployed  a  web-based  survey  using  the  SurveyMonkey  website. 
The  survey  instrument  included  both  demographic  questions  and  core  questions  related  to 
defining  and  measuring  successful  services  contracts.  The  core  questions  were  designed  to 
establish  the  importance  of  different  factors  when  defining  and  measuring  the  success  of 
services  contracts.  These  core  questions  were  related  to  the  contracting  process,  as  well  as 
to  different  outcomes  such  as  cost,  schedule,  and  performance  (Hagan  et  al. ,  2012). 

In  terms  of  defining  successful  contracts,  the  core  questions  asked  participants  to 
rank  various  definitions  relating  to  the  four  metrics  (process,  cost,  schedule,  and 
performance)  in  order  of  most  important  (1)  to  least  important  (5).  We  also  asked 
participants  to  rate  definition  statements  relating  to  process,  cost,  schedule,  and 
performance.  These  questions  use  a  Likert  scale  asking  level  of  agreement,  importance, 
and  amount  of  time  devoted  by  the  participants.  The  Likert  scale  had  a  range  of  1  to  5,  with 
1  representing  a  negative  response  and  5  representing  a  positive  response  (Hagan  et  al., 
2012). 

In  terms  of  measuring  successful  contracts,  the  core  questions  asked  participants  to 
rank  various  measurements  relating  to  the  four  metrics  in  order  of  most  important  (1)  to  least 
important  (5).  The  last  question  in  the  section  asks  participants  to  rate  on  a  Likert  scale  how 
often  the  organization  conducts  certain  actions  that  pertain  to  the  measurement  of  success 
concerning  process,  schedule,  cost,  and  performance.  Figure  2  reflects  our  survey  question 
approach  (Hagan  et  al.,  2012). 
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The  survey  was  deployed  to  the  major  stakeholders  (PMs,  COs,  and  CORs)  at  the 
following  major  contracting  commands:  Fleet  Logistics  Center  (FLC)  Philadelphia,  FLC 
Jacksonville,  FLC  Norfolk,  FLC  Puget  Sound,  FLC  San  Diego,  Naval  Sea  Systems 
Command  (NAVSEA),  Military  Sealift  Command  (MSC),  and  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR;  Hagan  et  al.,  2012). 

Survey  Results  and  Analysis 

In  this  section,  we  present  the  results  of  the  survey  and  discuss  its  major  findings.  As 
mentioned  previously,  the  primary  objective  of  this  research  is  to  empirically  examine  how 
the  success  of  a  service  contract  is  being  defined  and  measured  by  different  stakeholders. 
We  designed  a  survey  containing  19  questions  and  distributed  them  to  the  major 
stakeholders  in  the  services  acquisition  process  to  receive  their  responses.  The  survey  was 
deployed  at  the  eight  Navy  installations  identified  previously.  We  distributed  the  survey  to  a 
total  of  843  respondents  responsible  for  various  acquisition-related  functions.  Specifically, 
we  surveyed  the  following  stakeholders:  program  manager/project  officer,  contract 
officer/contract  specialist,  contracting  officer  representative,  requirements  manager,  financial 
manager,  contractor,  and  customer.  The  survey  questions  included  both  Likert-type  as  well 
as  ranking-type  questions.  The  Likert-type  questions  were  used  to  assess  favorable  or 
unfavorable  responses,  while  the  ranking-type  questions  were  used  to  assess  the  most 
important  responses.  When  we  examine  the  ranking  questions  in  this  section,  the  term 
“most  important”  refers  to  the  number  of  factors  that  received  the  highest  rankings  of  1  or  2. 
We  believe  that  this  is  the  best  way  to  capture  and  succinctly  represent  the  participants’ 
responses.  For  example,  a  COR  may  feel  that  the  outcome-related  factors  are  extremely 
important  and,  therefore,  should  be  given  the  highest  ranking  of  1  every  time.  However,  the 
COR  may  also  believe  that  the  process-related  factors  are  very  important,  too,  and  hence 
may  assign  the  next  highest  rank  of  2  to  those  factors.  Hence,  we  believe  that  the  percent  of 
respondents  giving  a  rank  of  1  or  2  to  a  factor  is  the  most  effective  way  to  capture  and 
represent  the  importance  of  that  factor  while  analyzing  the  data  on  ranking  of  factors. 

The  survey  response  rates  we  experienced  for  different  categories  of  stakeholders 
are  shown  in  Table  1 .  Unfortunately,  we  received  only  a  small  number  of  responses  from 
requirements  managers,  financial  managers,  contractors,  and  customers.  Hence,  their 
responses  are  not  incorporated  in  this  report  for  analysis  purposes.  These  respondents  are 
combined  under  the  “other”  category  in  Table  1 . 
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Table  1.  Survey  Response  Rate 


STAKEHOLDER 

#  SURVEYS 
DEPLOYED 

#  SURVEYS 
ANSWERED 

RESPONSE 

RATE 

PROGRAM  MANAGER/PROJECT  OFFICER 

94 

15 

16% 

CONTRACTING  OFFICER 

REPRESENTATIVE 

104 

27 

26% 

CONTRACTING  OFFICER/  CONTRACT 
SPECIALIST 

280 

126 

45% 

AGGREGATE  DATA  (PM,  COR,  PCO) 

478 

168 

35% 

OTHER 

365 

10 

2.7% 

TOTAL 

843 

178 

21% 

We  present  the  survey  results  and  analysis  in  three  sub-sections:  the  first  sub¬ 
section  presents  the  aggregate  data,  the  second  sub-section  presents  the  stakeholder-level 
data,  and  the  third  sub-section  presents  the  service-type  data. 


Survey  Results:  Aggregate  Survey  Data 

Defining  the  Success  of  a  Service  Contract 

In  taking  a  high-level  view  of  our  survey  findings,  we  did  not  differentiate  between 
functional  roles,  DAWIA  levels  of  certification,  type  of  service  being  acquired,  contract  type, 
or  the  organization.  However,  we  did  separate  our  findings  under  the  broad  categories  of 
process  and  outcome.  Outcome  results  included  the  questions  associated  with  cost, 
schedule,  and  performance.  As  shown  in  Table  1,  collectively,  there  were  168  responses 
from  PMs,  CORs,  or  PCOs.  The  Likert  scale  responses  were  assigned  a  value  of  1  through 
5,  with  the  higher  value  representing  a  more  favorable  response  to  a  statement.  A  summary 
of  aggregate  data  about  defining  and  measuring  the  success  of  a  service  contract  is 
presented  in  Tables  2  and  3  of  Appendix  A.  We  examined  the  mean  of  responses  to  each 
set  of  Likert  scale-type  questions.  We  found  that  when  defining  the  success  of  a  services 
contract,  outcomes  are  considered  slightly  more  important  than  processes.  The  overall 
mean  of  responses  related  to  outcomes  was  4.08,  while  process  responses  resulted  in  a 
mean  of  3.97.  Our  findings  are  displayed  graphically  in  Figure  3. 

We  then  separated  our  findings  further  within  the  broad  category  of  outcomes  into 
the  narrower  categories  of  cost,  schedule,  and  performance.  Performance-related  questions 
resulted  in  the  highest  mean  of  4.29,  while  cost-related  questions  produced  a  mean  of  4.03, 
and  schedule-related  questions  produced  a  mean  of  3.93. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors  related 
to  defining  the  success  of  a  service  contract.  These  questions  also  dealt  with  different 
aspects  of  processes  and  outcomes.  Of  the  168  respondents,  40%  felt  that  process-related 
factors  were  the  most  important.  Sixty  percent  felt  that  outcome-related  factors  were  the 
most  important.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  4. 

Breaking  down  the  outcome-related  factors  further,  15%  of  respondents  felt  that 
cost-related  factors  were  the  most  important,  19%  felt  that  schedule-related  factors  were 
most  important,  and  26%  felt  that  performance-related  factors  were  most  important. 
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Means  of  Aggregate  Stakeholder 
Definitions  of  Success  (n=168) 
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Figure  3.  Means  of  Aggregate  Stakeholder  Definitions  of  Success 
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Figure  4.  Aggregate  Stakeholder  Ranking  of  Definitions  of  Success 
Measuring  the  Success  of  a  Service  Contract 

Our  survey  also  requested  that  participants  rate  on  the  Likert  scale  the  various 
degrees  of  importance,  and  the  extent  to  which  they  agreed  or  disagreed  with,  various 
factors  when  considering  how  they  measure  the  success  of  a  service  contract.  Again,  these 
factors  related  to  either  processes  or  outcomes.  The  overall  Likert  scale  mean  with  relation 
to  processes  was  2.48,  and  the  outcomes  displayed  an  overall  mean  of  3.71.  Clearly 
outcomes  are  deemed  more  important  by  our  participants  as  a  whole.  Our  findings  are 
displayed  graphically  in  Figure  5. 
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If  we  look  at  the  distinct  factors  within  outcome  of  cost,  schedule,  and  performance, 
the  overall  Likert  means  were  3.96,  3.84,  and  3.30,  respectively. 

One  hundred  and  sixty-eight  respondents  were  asked  to  rank  different  factors  related 
to  measuring  the  success  of  a  service  contract.  Of  the  168  respondents,  46%  felt  that 
process-related  factors  were  the  most  important.  Fifty-four  percent  felt  that  outcome-related 
factors  were  the  most  important.  The  distribution  of  highest  ranked  responses  is  displayed  in 
Figure  6. 

Breaking  down  the  outcome-related  factors  further,  19%  of  respondents  felt  that 
cost-related  factors  were  the  most  important,  12%  felt  that  schedule-related  factors  were 
most  important,  and  23%  felt  that  performance-related  factors  were  most  important. 
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Figure  5.  Means  of  Aggregate  Stakeholder  Measurements  of  Success 
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Figure  6.  Aggregate  Stakeholder  Ranking  of  Measurements  of  Success 
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Analysis  of  Aggregate  Survey  Data 

The  findings  from  the  analysis  of  aggregate  survey  data  show  that  when  asked  to 
respond  on  a  Likert  scale,  different  stakeholders  find  all  aspects  of  processes  and  outcomes 
important  when  defining  the  success  of  a  service  contract.  The  means  of  the  responses  we 
collected  are  very  close,  and  it  does  not  seem  that,  as  a  whole,  our  population  favors 
process  or  outcome  when  defining  success.  Perhaps  this  is  due  to  the  nature  of  Likert  scale 
questions.  When  asked  if  something  such  as  cost  overruns,  major  milestones,  or  a  lack  of 
protests  is  important,  all  stakeholders  will  invariably  say  yes.  That  is  why  the  overall  mean  of 
all  responses,  for  both  outcomes  and  processes,  is  fairly  high  at  4.03.  When  forced  to  rank, 
the  responses  differ  and  outcome-related  responses  received  a  high  rank  of  1  or  2  60%  of 
the  time.  This  is  because  outcomes  such  as  keeping  on  schedule  and  budget  adherence  are 
easy  to  understand  and  define.  Process-related  factors  such  as  administration  and 
communication  are  relatively  harder  to  quantify. 

The  findings  also  demonstrate  that  when  measuring  the  success  of  a  service 
contract,  all  stakeholders  tend  to  focus  on  outcomes  and  do  not  take  into  consideration  the 
processes;  this  was  true  for  both  Likert-scale  responses  and  ranking  responses.  This  is  very 
evident  in  the  Likert-scale  responses,  where  none  of  the  process-related  factors  showed  a 
mean  of  3  or  more.  When  forced  to  rank  the  different  factors  with  respect  to  measuring 
success,  the  results  were  similar  to  defining  success,  with  56%  of  “most  important” 
responses  falling  under  the  outcomes  category. 

In  general  our  findings  from  the  “other”  category  mirrored  our  aggregate  results. 
Although  there  were  only  10  responses,  all  felt  that  outcomes  were  the  most  important  factor 
when  defining  and  measuring  the  success  of  a  service  contract.  We  found  that  our 
stakeholders  in  this  category  rated  and  ranked  processes  extremely  low  in  both  defining  and 
measuring  the  success  of  a  service  contract.  This  is  because  these  stakeholders  are  not 
terribly  burdened  by  administration  and  other  process-related  factors,  so  they  feel  that  these 
factors  are  not  important.  For  example,  a  contractor  or  end  user  does  not  necessarily 
conduct  market  research  or  choose  the  appropriate  contract  type.  However,  they  are  very 
concerned  with  staying  within  cost,  keeping  up  with  schedule,  and  maintaining  a  high  level 
of  performance. 

Survey  Results:  Stakeholder-Level  Data 

As  a  starting  point  in  examining  how  different  stakeholders  define  and  measure  the 
success  of  a  service  contract,  we  performed  a  statistical  analysis  of  the  data  to  determine 
whether  there  were  significant  differences  between  the  ratings  on  the  Likert  scale  across  the 
major  stakeholders.  We  first  performed  an  F-test  for  sample  variances  to  determine  the 
appropriate  f-test  to  perform.  In  all  instances,  we  found  that  there  was  an  equal  variance 
among  stakeholders.  The  only  statistically  significant  difference  was  between  the  CORs  and 
COs/specialists  when  measuring  success.  This  could  be  due  to  the  fact  that  CORs  view 
communication  and  other  processes  as  key  factors  when  measuring  the  success  of  a 
service  contract.  The  COR  is  also  likely  to  view  a  protest  as  a  serious  issue  when  measuring 
success  because  it  results  in  a  delay  of  execution  and  CORs  cannot  perform  their  duties. 
Otherwise,  there  was  no  statistically  significant  difference  between  any  other  of  the 
stakeholders  on  the  Likert  scale.  We  discuss  in  the  next  section  the  results  of  the  analysis  of 
stakeholder-level  data. 

Analysis  of  Stakeholder-Level  Data 

Consistent  with  the  abovementioned  results  of  statistical  analysis,  we  found  that 
PMs,  CORs,  COs,  and  contract  specialists  all  agree  that  outcome  is  slightly  more  important 
than  processes  based  on  participants’  ratings  of  separate  factors  on  a  Likert  scale.  Each 
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functional  role  rated  outcome  slightly  over  4.00,  while  rating  processes  just  below  4.00.  The 
mean  of  the  functional  roles  combined  was  3.94  for  processes,  and  4.1 1  for  outcomes. 
Within  outcome,  performance-related  factors  received  the  highest  average  rating,  while 
schedule-related  factors  received  the  lowest  average  rating.  All  functional  roles  showed  an 
upward  trend  from  schedule,  to  cost,  to  performance.  A  comparison  of  our  Likert  scale 
findings  for  defining  success  across  functional  roles  is  displayed  graphically  in  Figure  7. 

When  stakeholders  were  asked  to  rank  different  factors  concerning  their  definition  of 
success,  we  found  that  there  was  clear  agreement  that  outcomes  are  more  important  than 
processes.  There  was,  however,  some  disagreement  within  the  outcome  factors  of  cost, 
schedule,  and  performance.  CORs  felt  that  cost  was  the  most  important  factor,  while  PMs, 
COs,  and  specialists  placed  performance  at  the  top  of  their  rankings.  Examined  collectively, 
the  major  stakeholders  provided  1 68  responses  when  ranking  their  definition  of  the  success 
of  a  service  contract.  Sixty  percent  of  respondents  felt  that  outcome-related  factors  were 
most  important,  while  40%  felt  that  process-related  factors  were  the  most  important  when 
defining  success.  The  distribution  of  highest  ranked  responses  is  displayed  in  Figure  8. 
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Figure  7.  Definitions  of  Success  Across  Major  Stakeholders 
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Figure  8.  Major  Stakeholder  Ranking  of  Definitions  of  Success 


According  to  the  survey  data,  stakeholders  also  tend  to  measure  success  in  almost 
the  same  way.  When  asked  to  rate  different  factors  on  the  Likert  scale  related  to 
stakeholders’  measures  of  success,  all  respondents  agreed  that  outcomes  far  outweigh 
processes.  When  looking  at  the  mean  across  stakeholders,  processes  received  a  rating  of 
2.56,  while  outcomes  received  a  rating  of  3.78.  Within  outcome-related  factors,  stakeholders 
showed  an  upward  trend  from  performance,  to  schedule,  to  cost.  A  comparison  of  our 
findings  for  defining  success  on  the  Likert  scale  across  functional  roles  is  displayed 
graphically  in  Figure  9. 

Our  ranking  data  shows  that,  again,  major  stakeholders  prefer  outcome-related 
factors  when  measuring  the  success  of  service  contracts.  When  examined  in  aggregate,  the 
major  stakeholders  provided  168  responses  to  our  ranking  questions.  Of  these  responses, 
43%  of  respondents  felt  process  factors  were  most  important,  while  57%  were  in  favor  of 
factors  related  to  outcomes.  The  distribution  of  highest  ranked  responses  is  displayed  in 
Figure  10. 
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Figure  9.  Measurement  of  Success  Across  Major  Stakeholders 
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Figure  10.  Major  Stakeholder  Ranking  of  Measurements  of  Success 

The  Likert  scale  responses  for  definitions  of  success  were,  again,  relatively  high,  and 
this  was  due  to  the  reason  explained  earlier.  It  is  interesting  that  in  both  defining  and 
measuring  success,  CORs  ranked  cost  highest  out  of  the  three  stakeholders. 

Another  interesting  result  is  that  COs  tended  to  place  nearly  equal  importance  on 
process  and  outcomes  when  forced  to  rank  factors  concerning  measuring  success.  This  is 
probably  due  to  the  administrative  nature  of  the  COs’  role.  For  example,  their  functional  role 
has  to  deal  with  modifications,  COR  reports,  and  exercising  options.  The  other  functional 
roles  of  PMs  and  CORs  are  not  overly  concerned  with  processes  and  are  focused  on  the 
requirement  and  outcomes.  The  data  reflect  this  fact. 
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It  is  interesting  to  note  that  every  demographic  consistently  rated  processes 
significantly  higher  on  the  Likert  scale  when  defining  success  versus  measuring  success. 

We  feel  that  this  is  because  stakeholders  view  measures  as  a  tangible  entity  associated  with 
post-award  functions.  Measures  such  as  cost,  schedule,  and  performance  are  fairly 
straightforward  inasmuch  as  a  goal  is  either  met  or  not.  Processes  such  as  communication 
flow  and  overall  management  are  more  obscure  and  subjective.  The  stakeholders  rated 
processes  higher  for  defining  success  because  they  are  closely  associated  with  mainly  pre¬ 
award  functions.  Processes  such  as  choosing  the  correct  contract  type  and  appropriately 
evaluating  the  proposal  are  crucial  for  success.  Because  these  are  pre-award  activities,  it  is 
easier  to  define  success  rather  than  measure  it. 

Summary,  Conclusions,  and  Recommendations 

Summary 

The  DoD’s  obligations  on  contracts  have  more  than  doubled  between  fiscal  years 
2001  and  2008  to  over  $387  billion,  with  over  $200  billion  spent  just  for  services  in  2008 
(GAO,  2009).  In  conjunction  with  this  increase  in  defense  procurement  is  the  reduction  of 
the  defense  acquisition  workforce.  The  combination  of  the  increasing  defense  procurement 
workload  and  the  decreasing  size  of  the  government  workforce,  along  with  the  complexities 
of  an  arcane  and  convoluted  government  contracting  process,  have  created  the  perfect 
storm — an  environment  in  which  complying  with  government  contracting  policies  and 
adopting  contract  management  best  practices  has  not  always  been  feasible  (Rendon,  2010). 
The  contract  management  process  is  performed  with  inputs  from  the  different  functional 
areas,  using  a  cross-functional  team  or  integrated  project  team  (IPT)  structure.  Each  of 
these  project  team  members  represents  the  stakeholders,  and  their  different  goals  and 
objectives.  The  first  research  question  we  investigated  was  as  follows:  How  do  different 
stakeholders  define  successful  services  contracts  within  the  Navy?  To  develop  a  clear 
understanding  of  current  services  acquisition  practices,  we  also  investigated  a  second 
research  question:  How  do  different  stakeholders  measure  services  contracts  within  the 
Navy?  Investigating  the  above  two  questions  helped  us  develop  recommendations 
regarding  the  third  and  final  research  question:  How  should  the  service  contract’s  success 
be  measured? 

Conclusions 

On  the  aggregate  level,  our  research  indicated  that,  when  defining  a  successful 
service  contract,  stakeholders  considered  outcomes  (in  the  order  of  performance,  cost,  and 
schedule)  slightly  more  important  than  processes.  Stakeholders  also  ranked  outcome- 
related  factors  as  most  important.  On  the  aggregate,  our  research  indicated  that,  when 
measuring  a  successful  service  contract,  stakeholders  considered  outcomes  (in  the  order  of 
cost,  schedule,  and  performance)  more  important  than  processes.  Stakeholders  also  ranked 
outcome-related  factors  as  most  important. 

On  the  stakeholder  level,  our  research  indicated  that,  when  defining  a  successful 
service  contract,  PMs,  CORs,  and  COs  considered  outcomes  (in  the  order  of  performance, 
cost,  and  schedule)  slightly  more  important  than  processes.  PMs,  CORs,  and  COs  also 
ranked  outcome-related  factors  as  most  important.  On  the  stakeholder  level,  our  research 
indicated  that,  when  measuring  a  successful  service  contract,  PMs,  CORs,  and  COs 
considered  outcomes  (in  the  order  of  performance,  schedule,  and  cost)  more  important  than 
processes.  PMs,  CORs,  and  COs  also  ranked  outcome-related  factors  as  most  important. 

Recommendations 

Our  research  findings  have  several  implications  for  the  Navy,  as  well  as  the  DoD.  All 
stakeholders  surveyed  identified  and  ranked  outcome-related  factors  as  more  important 
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than  process-related  factors,  in  both  defining  and  measuring  the  success  of  service 
contracts.  This  may  be  because  outcome-related  factors  (cost,  schedule,  and  performance) 
are  more  easily  defined  and  measured  using  available  metrics,  compared  to  contracting 
processes,  which  are  more  difficult  to  define,  and  many  agencies  have  no  available  metrics. 
However,  as  discussed  in  the  earlier  sections  of  this  paper,  many  of  the  contracting 
deficiencies  identified  by  the  GAO  and  DoDIG  are  related  to  contracting  processes,  such  as 
conducting  market  research,  determining  item  commerciality,  selecting  contract  type, 
negotiating  fair  and  reasonable  prices,  and  monitoring  contractors  through  surveillance. 
Thus,  our  first  recommendation  is  that  the  U.S.  Navy  develop  and  implement  process- 
related  metrics  to  define  and  measure  critical  contracting  processes,  such  as  conducting 
market  research,  determining  item  commerciality,  selecting  contract  type,  negotiating  fair 
and  reasonable  prices,  and  monitoring  contractors. 

Our  literature  review  identified  that  acquisition  stakeholders  (PMs,  CORs,  and  COs) 
have  different  procurement  goals  and  objectives,  and  these  goals  and  objectives  may  in  fact 
conflict  with  each  other.  Our  second  recommendation  is  that  the  U.S.  Navy  should  establish 
internal  controls  to  ensure  the  contracting  processes  are  being  followed  and  that  the 
different  stakeholders  place  sufficient  importance  on  the  value  of  these  contracting 
processes. 

Finally,  as  previous  research  has  determined  that  contracts  are  only  as  successful  as 
the  processes  used  to  plan,  award,  and  administer  these  contracts,  our  final 
recommendation  is  for  the  U.S.  Navy  to  implement  a  program  for  continuously  assessing  its 
contracting  process  capability  and  using  the  assessment  results  to  improve  its 
organizational  contract  management  process  capability.  Once  the  U.S.  Navy,  as  well  as  the 
DoD,  implement  contracting  process-related  metrics  to  define  and  measure  services 
contracts,  internal  controls  to  ensure  contracting  process  compliance,  and  periodical 
assessments  of  organizational  contracting  process  capability,  the  importance  of  process- 
related  factors  in  defining  and  measuring  the  success  of  service  contracts  will  increase 
among  stakeholders  and  thus  start  addressing  some  of  the  contracting  deficiencies 
identified  by  the  GAO  and  the  DoDIG. 
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Abstract 

Over  the  last  decade,  Department  of  Defense  (DoD)  spending  on  service  contracts  more  than 
doubled  in  constant  terms,  from  $90  billion  in  2000  to  $183  billion  in  2012.  Policy  makers 
have  recently  attempted  to  reduce  or  even  reverse  this  trend,  in  part  by  emphasizing  instead 
the  “in-sourcing”  of  work  performed  under  services  contracts.  Over  the  last  three  years,  the 
Center  for  Strategic  and  International  Studies  (CSIS)  has  worked  to  develop  a  more 
systematic  framework  for  guiding  sourcing  decisions  for  services  contracts  within  the  DoD, 
which  would  have  broader  implications  for  the  whole  universe  of  budget-based  decisions 
within  the  DoD.  Towards  that  purpose,  this  paper  analyzes  the  stated  motivations, 
implementation  strategies,  and  guiding  analytical  underpinnings  for  previous  outsourcing 
efforts  and  for  the  currently  ongoing  in-sourcing  initiative.  It  then  assesses  current  and 
previous  DoD  methodologies  for  guiding  sourcing  decisions,  highlighting  the  individual 
strengths  and  shortcomings  of  these  methodologies.  The  third  section  of  this  paper  analyzes 
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public  sector  sourcing  decisions  in  the  wider  context  of  economics  and  business 
management,  to  provide  broader  conceptual  insights  for  more  informed  determinations  on 
these  sourcing  decisions.  All  of  this  research  is  being  used  to  develop  a  repeatable, 
verifiable,  data-driven  methodology  to  guide  sourcing  decisions,  which  will  be  presented  in 
the  final  report  of  this  project. 

Introduction 

Over  the  last  decade,  Department  of  Defense  (DoD)  spending  on  service  contracts 
more  than  doubled  in  constant  terms,  from  $90  billion  in  2000  to  $183  billion  in  201 2. 1  Policy 
makers  have  recently  attempted  to  reduce  or  even  reverse  this  trend,  in  part  by  emphasizing 
instead  the  “in-sourcing”  of  services  contracts.  In  the  past,  conversions  from  government 
civilians  to  contractors  have  been  done  for  reasons  of  policy  or  projected  cost  savings.  More 
recently,  conversions  from  contractors  to  government  civilians,  as  well  as  other  actions  to 
expand  the  federal  workforce,  have  been  undertaken  for  a  similar  combination  of  policy 
reasons  and  projected  cost  savings.  Weaknesses  in  the  methodology  used  by  the  DoD  to 
justify  or  budget  for  in-sourcing  decisions  call  into  question  whether  the  DoD  is  using 
accurate  data  on  the  cost  implications  of  its  sourcing  decisions. 

Over  the  last  three  years,  the  Center  for  Strategic  and  International  Studies  (CSIS) 
has  worked  to  develop  a  more  systematic  framework  for  guiding  sourcing  decisions  for 
services  contracts  within  the  DoD.  This  framework  also  has  broader  implications  for  all 
budget-based  decisions  within  the  DoD.  Towards  that  purpose,  this  paper  first  analyzes  the 
stated  motivations,  implementation  strategies,  and  guiding  analytical  underpinnings  for 
previous  outsourcing  efforts  and  for  the  currently  ongoing  in-sourcing  initiative.  It  then 
assesses  current  and  previous  DoD  methodologies  for  guiding  sourcing  decisions, 
highlighting  the  individual  strengths  and  shortcomings  of  these  methodologies.  The  third 
section  of  this  paper  analyzes  public  sector  sourcing  decisions  in  the  wider  context  of 
economics  and  business  management,  to  provide  broader  conceptual  insights  for  more 
informed  sourcing  decisions.  All  of  this  research  is  being  designed  to  support  the 
development  of  a  repeatable,  verifiable,  data-driven  methodology  to  guide  sourcing 
decisions,  which  will  be  presented  in  the  final  report  of  this  project. 

Department  of  Defense  Sourcing  Policy 

Office  of  Management  and  Budget  (OMB)  Circular  A-76 

OMB  Circular  A-76  was  the  result  of  over  three  decades  of  policy  deliberation 
towards  ensuring  that  the  government  did  not  improperly  compete  with  private  enterprise. 
Starting  in  the  1930s,  a  series  of  commissions  and  reports  grappled  with  the  problem  of 
what  tasks  should  (or  must)  be  performed  by  government  employees,  and  what  tasks  are 
better  left  to  the  private  sector.  These  debates  culminated  during  the  1950s  and  1960s  in 
the  issuing  of  guidance  documents  that  ultimately  became  Circular  A-76  (hereafter  referred 
to  as  “A-76”),  which  sought  to  lay  out  uniform  guidance  on  sourcing  policy  across  the  federal 
government2  (Halchin,  2007,  pp.  3-4). 

A-76  has  been  revised  several  times  since  its  issuance,  but  the  core  of  the  guidance 
has  always  been  the  competitive  process,  better  known  as  public-private  competition.  A-76 
has  never  mandated  competition  for  any  particular  function  (though  two  administrations, 
those  of  Presidents  Ronald  Reagan  and  George  W.  Bush,  issued  policies  setting  targets  for 


1  Federal  Procurement  Data  System  (FPDS.gov)  data  with  CSIS  analysis. 

2  The  most  recent  revision  of  Circular  A-76,  issued  in  2003,  can  be  viewed  here: 

http://www.whitehouse.gov/sites/default/files/omb/assets/omb/circulars/a076/a76_incl_tech_correctio 

n.pdf 
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numbers  of  positions  to  be  competed);  rather,  A-76  laid  out  procedures  for  how  such  public- 
private  competitions  were  to  be  conducted  (Halchin,  2007,  p.  6).  The  competitive  process 
included  three  broad  steps,  once  a  function  had  been  identified  for  competition: 

1 .  issuance  of  a  Performance  Work  Statement,  to  lay  out  clearly  the  tasks  to  be 
performed  and  ensure  that  competitors  were  “bidding”  for  the  same  work; 

2.  formation  of  a  Most  Efficient  Organization  (MEO)  within  the  government  to 
serve  as  the  government’s  offeror;  and 

3.  selection  of  a  private  competitor  from  the  field  of  bidders,  to  compare  against 
the  government  option. 

After  adjustments  to  compensate  for  differences  in  projected  performance  levels,  to 
ensure  balanced  and  fair  cost  comparisons,  if  the  private  bid  were  10%  or  $10  million  less 
than  the  government  option,  the  function  would  be  outsourced  (Commercial  Activities  Panel 
[CAP],  2002,  p.  19). 

OMB  Circular  A-76  Within  the  DoD 

From  the  start,  the  DoD  has  been  the  most  active  agency  in  performing  A-76  cost 
comparisons.  After  increasing  sharply  in  the  late  1970s  and  early  1980s,  A-76  competitions 
within  the  DoD  declined  by  over  half  in  the  latter  half  of  the  1980s,  a  trend  which  continued 
into  the  early-  to  mid-1990s,  when  very  few  competitions  were  started  (Keating,  1997,  p.  4). 
Competitions  started  to  increase  in  the  late  1990s  and  early  2000s,  but  between  1997  and 
2001 ,  there  were  fewer  cost  comparisons  performed,  combined,  than  in  any  individual  year 
in  the  early  1980s  (CAP,  2002,  p.  21).  In  the  late  1990s  and  early  2000s,  A-76  was  one  part 
of  the  DoD’s  comprehensive  “strategic  sourcing”  initiatives,  designed  to  cover  the  whole 
range  of  DoD  activities  (GAO,  2000,  p.  3).  Historically,  the  Navy  (which  has  conducted  the 
most  competitions)  and  Air  Force  have  had  the  most  active  A-76  cost-comparison  programs, 
with  the  Army  conducting  about  a  third  fewer  competitions  than  the  Navy,  and  the  USMC 
and  various  DoD  agencies  each  accounting  for  less  than  a  sixth  of  the  total  number  of 
competitions  started  by  the  Navy  (Keating,  1997,  p.  7). 

Numerous  studies  have  shown  that  the  A-76  competitions  have  produced  significant 
savings,  more  as  a  result  of  competitive  pressures  than  any  inherent  advantage  of  public  or 
private  providers  (Tighe  et  al.,  1996,  p.  11).  The  government  MEOs  and  industry  each  won 
approximately  half  of  the  competitions,  on  average  (Keating,  1997,  p.  18).  A  review  of 
several  studies  on  savings  produced  through  A-76  competitions  showed  an  average  savings 
of  around  30%  across  a  number  of  different  functions  and  tasks,  though  that  number  was 
highly  variable  (ranging  from  15%  to  45%).  One  study  noted  that  the  highest  savings  were 
achieved  when  military  billets  were  converted,  though  there  are  limits  to  which  military 
functions  can  be  classified  as  “commercial”  or  not  inherently  governmental. 

Criticisms  and  Problems  With  A-76  Implementation 

In  reviewing  the  literature,  the  majority  of  technical  criticisms  of  the  A-76  process 
focus  not  on  the  policy  itself  but  rather  on  the  implementation  of  the  competitions.  One 
particularly  troubling  figure  is  seen  in  a  RAND  review  of  DoD  A-76  cost  comparisons:  For 
every  13  cost  comparisons  started  in  the  period  reviewed,  five  were  cancelled  (Keating, 
1997,  p.  9).  These  cancellations  happened  for  a  number  of  reasons,  though  large  delays  in 
soliciting  and  preparing  bids  seemed  to  be  a  common  cause,  and  studies  of  large  functions 
were  at  greater  risk  of  being  cancelled  before  completion.  A  provision  in  the  fiscal  year  (FY) 
1991  DoD  Appropriations  Act  imposing  a  24-month  limit  on  single-function  cost  comparisons 
going  forward  also  influenced  the  rate  of  cancellations  (Keating,  1997,  p.  x).  The  length  of 
time  for  competitions  to  be  completed  was  a  recurring  problem  cited  in  the  literature; 
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according  to  the  aforementioned  RAND  study,  the  median  time  for  completion  was  664 
days,  with  a  mean  of  810  days  (Keating,  1997,  p.  35).  In  discussions  with  stakeholders,  the 
long  delays  were  seen  as  troublesome  by  both  industry  and  government  sources,  due  to 
morale  issues  caused  by  uncertainty  regarding  job  security  (on  the  government  side)  and 
the  inability  to  plan  revenues  and  workload  (on  the  industry  side.) 

The  lack  of  post-decision  follow-up  on  A-76  competitions  was  another  major  source 
of  criticism.  Despite  some  mechanisms  in  place,  there  was  no  consistent  effort  within  the 
DoD  to  track  whether  A-76  competitions  produced  projected  savings  or  met  promised 
performance  levels  (GAO,  2002,  p.  4).  Another  major  area  of  criticism  was  with  how  A-76 
was  being  used.  The  2002  Commercial  Activities  Panel  report  evaluating  government 
sourcing  policy  noted  that,  while  A-76  functioned  reasonably  well  as  a  way  to  compare  the 
cost  of  government  and  private  performance,  it  was  being  stretched  to  include  evaluations  of 
other  factors  it  was  never  designed  to  weigh:  “quality,  innovation,  flexibility,  and  reliability” 
(CAP,  2002,  pp.  10,  41-43). 

Moratorium 

In  January  2008,  Congress  passed  legislation  suspending  A-76  cost  competitions 
within  the  DoD  (and  throughout  the  rest  of  the  government  in  March  2009),  a  prohibition 
which  has  been  consistently  renewed  in  the  years  since  then.  Attached  to  legislation 
continuing  the  prohibitions  in  201 0  and  201 1  were  calls  for  studies  of  A-76  to  be  completed 
by  various  stakeholders,  including  the  DoD  Inspector  General  (DoDIG)  and  the  Government 
Accountability  Office  (GAO),  which  would  be  used  to  determine  whether  A-76  competitions 
would  be  allowed  to  resume.  Although  all  required  studies  have  been  delivered  to  Congress, 
and  many  of  them  recommend  resuming  A-76  competitions,  neither  Congress  nor  the 
current  administration  have  acted  to  revive  A-76.  In  fact,  the  President’s  FY2013  budget 
request  includes  a  provision  explicitly  prohibiting  funds  from  being  used  for  any  outsourcing- 
related  study  or  competition  (Bailey  Grasso,  2013,  pp.  5-8). 

The  DoD’s  In-Sourcing  Initiative 

On  April  6,  2009,  Secretary  of  Defense  Robert  M.  Gates  announced  a  plan  to  reduce 
the  DoD’s  reliance  on  contractors  and  expand  its  use  of  federal  civilians  to  provide  services 
(Gates,  2009).  Between  2010  and  2015,  this  in-sourcing  initiative  projected  the  replacement 
of  more  than  30,000  contractors  with  DoD  civilians.  According  to  Gates’  announcement,  this 
would  “restore  balance”  to  the  workforce  by  returning  the  ratio  of  contractors  to  DoD  civilians 
to  its  2001  level.  A  plain  reading  of  contemporaneous  budget  documents  indicates  that  the 
plan  was  also  based  on  an  assumption  that  federal  civilians  would  be  significantly  less 
costly  than  the  contractors  they  replaced.  As  a  result,  the  DoD  planned  to  achieve  budgetary 
savings  equal  to  40%  of  the  cost  of  the  contractors  being  replaced;  more  recent  DoD 
statements  claimed  savings  of  25%  (Gates,  2010).  Neither  figure  appears  justifiable — 
research  has  shown  that  the  about  65%  any  savings  achieved  through  public-private 
competitions  derive  from  the  competition  itself,  not  from  any  intrinsic  advantage  on  either  the 
public  or  private  side.3 The  FY2010  DoD  budget  reflected  those  savings,  as  have 
subsequent  DoD  budget  proposals  to  Congress. 

This  initiative  was  consistent  with  a  variety  of  other  legislative  and  policy  decisions 
on  the  role  of  government  contractors.  The  National  Defense  Authorization  Acts  (NDAAs)  of 
2006  and  2008  required  the  DoD  to  consider  greater  use  of  federal  civilians.  A  March  4, 
2009,  Presidential  Memorandum  on  government  contracting  required  the  OMB  to  review 


3  See,  for  example,  Snyder,  Trost,  and  Trunkey  (1998)  and  Trunkey,  Trost,  and  Snyder  (1996). 
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policies  for  contracting  for  services  (Obama,  2009).  Numerous  GAO  and  DoD  IG  reports 
have  cited  the  DoD’s  over-reliance  on  contractors.4 

A  DoD  report  to  Congress  in  December  2009  indicated  that  17,000  additional  civilian 
positions  would  be  established  in  2010  as  the  result  of  new  in-sourcing  efforts  (McGinn, 
2009,  p.  6).  Of  this  17,000,  half  are  for  commercial  activities,  which  the  report  states  can  be 
done  at  lower  cost  in-house.  Another  42%  are  for  commercial  activities  that  the  DoD  would 
exempt  from  private  sector  performance  on  the  grounds  that  they  support  readiness  or 
workforce  management  needs,  including  the  need  to  provide  for  career  progression  and  for 
the  “oversight  and  control  of  functions  closely  associated  with  inherently  governmental  work” 
(McGinn,  2009,  p.  5).  The  remaining  8%  is  for  work  that  the  DoD  has  determined  is 
inherently  governmental.  The  reliance  on  cost  analysis  for  half  of  the  in-sourcing  goals 
clearly  puts  a  burden  on  the  DoD  using  proper  taxonomies  and  methodologies  to  compare 
the  cost  of  government  employees  and  contractors  (McGinn,  2009,  pp.  4-5). 

The  December  2009  DoD  report  included  a  number  of  changes  from  the  plans 
announced  in  April  2009.  One  significant  change  was  to  expand  the  types  of  services 
affected  by  the  initiative.  The  original  plan  focused  on  two  budget  categories — advisory 
assistance  services  and  the  category  called  “other  services.”  However,  that  plan  was 
expanded  to  allow  managers  to  consider  any  type  of  contracted  service  for  in-sourcing, 
including  activities  such  as  laundry  services,  installation  maintenance,  and  transportation. 
Targeting  these  expanded  activities  for  in-sourcing  is  only  consistent  with  previous  policy 
directives  if  cost  savings  can  be  realized.  CSIS  concluded  at  the  time  that  the  process  was 
insufficient  to  validate  those  savings  and  that  there  were  sound  reasons  to  suspect  they 
would  not  be  achieved  (Berteau  et  al. ,  201 1 ,  pp.  5-7). 

In  an  August  9,  2010,  statement,  Secretary  of  Defense  Gates  himself  de-emphasized 
in-sourcing,  signaling  that  expected  savings  were  not  materializing.  Subsequent  statements 
from  DoD  officials  have  stated  that  existing  in-sourcing  initiatives  by  the  military  departments 
remain  in  full  force,  however  (Brodsky,  2010).  In  the  course  of  this  research  effort, 
discussions  with  DoD  officials  have  indicated  that  the  expected  savings  from  in-sourcing  are 
still  built  into  budgets,  and  some  within  the  DoD  still  believe  that  in-sourcing,  in  and  of  itself, 
will  lead  directly  to  large  savings.  The  Ike  Skelton  National  Defense  Authorization  Act  for 
Fiscal  Year  201 1  mandates  that  the  “Secretary  of  Defense  shall  use  the  costing 
methodology  outlined  in  the  Directive-Type  Memorandum  09-007  (Estimating  and 
Comparing  the  Full  Costs  of  Civilian  and  Military  Manpower  and  Contractor  Support)  or  any 
successor  guidance  for  the  determination  of  costs  when  costs  are  the  sole  basis  for  the 
decision.” 

Recent  legislative  action  and  statements  from  the  DoD  do  show  a  weakening  of 
support  for  in-sourcing.  Secretary  of  the  Army  John  McHugh  suspended  all  of  the  Army’s  in¬ 
sourcing  activities  through  a  February  1, 2011,  memorandum  on  “Reservation  of  In-Sourcing 
Approval  Authority.”  More  recently,  section  937  of  H.R.  1540,  the  House  version  of  the 
FY2012  National  Defense  Authorization  Act,  called  for  an  end  to  the  temporary  moratorium 
on  public-private  competitions  that  was  established  in  the  FY2010  NDAA.  Though  this 
provision  did  not  make  it  into  the  final  bill,  it  does  signal  a  shift  in  Congressional  support 
away  from  in-sourcing. 

The  release  of  the  Office  of  Federal  Procurement  Policy’s  Policy  Letter  11-01, 
released  on  September  1 2,  201 1 ,  marks  the  most  recent  major  policy  development  relating 
to  the  broader  issue  of  sourcing  decisions.  This  guidance  provides  a  much-needed 


4  See,  for  example,  GAO  (2006)  and  DoD  Inspector  General  (2009). 
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framework  for  sourcing  decisions  based  on  three  categories  of  work:  inherently 
governmental,  closely  associated,  and  critical  classifications.  While  this  guidance  represents 
a  welcome  step  in  the  right  direction  towards  clarifying  the  standards  for  declaring  positions 
or  functions  inherently  governmental  or  closely  associated,  various  stakeholders  have 
expressed  a  desire  for  more  specific  guidance  going  forward  to  help  eliminate  uncertainty 
regarding  the  boundaries  of  those  categories,  and  the  guidance  for  “critical  classifications” 
has  also  been  called  ambiguous  and  imprecise. 

DoD  Cost-Estimation  Methodologies 

Given  that  the  focus  of  DoD  sourcing  policy  has  been  on  issues  of  cost,  the 
soundness  of  the  cost-estimation  methodologies  at  the  heart  of  those  policies  is  crucial.  As 
CSIS  has  noted  in  previous  work  on  the  subject,  however,  having  a  repeatable,  verifiable, 
data-driven  cost-estimation  methodology  for  calculating  the  cost  of  government  performance 
is  critical  even  outside  the  realm  of  sourcing  policy.  Particularly  in  times  of  budgetary  strain 
such  as  exist  today,  the  DoD  will  be  making  decisions  about  the  future  of  programs  and 
functions  based  on  perceived  potential  cost  savings.  Without  a  rigorous  cost-estimating 
methodology  to  determine  the  fully  burdened  cost  of  a  particular  function  to  the  government 
as  a  whole  (or  even  simply  to  the  department),  the  DoD  lacks  a  process  to  gather,  verify, 
and  use  the  data  it  needs  to  make  such  decisions,  without  which  it  will  not  know  the  true 
cost  implications  of  the  decisions  it  makes. 

Since  2009,  the  cost-estimating  methodology  of  DTM  09-007  has  replaced  the 
methodology  from  A-76  as  the  standard  for  use  within  the  DoD.  As  has  been  explored  in 
previous  work  by  CSIS  on  the  subject,  this  change  did  not  represent  an  improvement 
(Berteau  et  al.,  201 1 ,  pp.  9-1 1 ). 

Directive-Type  Memorandum  09-007 

In-sourcing  decisions  made  on  the  basis  of  cost  depend  on  the  ability  to  project 
accurately  the  relative  costs  of  the  governmental  and  private  options.  Further,  even  if  in¬ 
sourcing  is  done  for  policy  reasons  (such  as  rebuilding  the  DoD  acquisition  work  force),  the 
DoD  still  needs  to  know  the  cost  impact  of  these  actions.  Without  these  data,  any  cost 
comparison  is  no  more  than  guesswork.  In  part  to  meet  those  objectives,  on  January  29, 
2010,  the  Director  of  the  Cost  Analysis  and  Program  Evaluation  (CAPE)  signed  Directive- 
Type  Memorandum  (DTM)  09-007,  “Estimating  and  Comparing  the  Full  Costs  of  Civilian  and 
Military  Manpower  and  Contract  Support.”  This  DTM  constitutes  current  DoD  guidance  for 
in-sourcing  decisions,  and  the  NDAA  for  Fiscal  Year  201 1  mandates  that  the 

Secretary  of  Defense  shall  use  the  costing  methodology  outlined  in  the 
Directive-Type  Memorandum  09-007  (Estimating  and  Comparing  the  Full 
Costs  of  Civilian  and  Military  Manpower  and  Contractor  Support)  or  any 
successor  guidance  for  the  determination  of  costs  when  costs  are  the  sole 
basis  for  the  decision. 

Yet  the  procedures  laid  out  in  the  DTM  for  calculating  the  government’s  costs  for 
performing  a  service  have  several  significant  gaps.  These  gaps  raise  questions  about  the 
validity  of  any  analysis  generated  on  the  basis  of  DTM  guidance.  The  DTM  is  written  to 
encourage  analysts  to  “carefully  consider”  all  possible  costs  associated  with  contracts,  but 
the  guidance  itself  overlooks  many  cost  aspects  for  the  government  side.  Among  other 
shortfalls,  the  DTM 

•  Lacks  the  ability  to  calculate  fully  burdened  government-wide  costs.  The  DTM 
states  that  “manpower  cost  estimates  normally  address  costs  to  the 
Department  of  Defense,”  and  that  “the  costs  of  service  contracts  are  variable 
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costs  in  the  short  run  paid  by  the  Department  of  Defense.”  Analysts  have 
interpreted  the  lack  of  consistent  focus  on  fully  burdened  government-wide 
costs  to  mean  they  could  leave  out  costs  or  savings  that  accrue  not  to  the 
DoD  but  to  other  federal  agencies. 

•  Creates  a  gap  by  failing  to  account  for  the  full  cost  of  DoD-owned  capital 
while  requiring  the  inclusion  of  those  costs  for  contractors.  This  ignores  the 
fact  that  the  real  economic  costs  of  capital  devoted  to  risky  commercial 
activities — including  forgone  interest  and  a  risk  premium  as  well  as 
depreciation — are  present  regardless  of  whether  the  activity  is  performed  by 
a  public  or  private  producer.  The  failure  to  consider  any  capital  costs  for 
government  workers  is  a  step  backwards  from  the  costing  approach  used 
under  OMB  Circular  A-76  (see  the  following  section),  which  included  the  cost 
of  in-house  production  at  a  private  sector  rate  of  return  on  new  investments.  It 
is  difficult  to  determine  the  federal  cost  of  capital,  but  there  is  universal 
agreement  that  the  cost  is  not  zero. 

•  Fails  to  account  for  taxes  forgone  by  the  federal  treasury  or  state  or  local 
governments.  This  is  another  step  back  from  OMB  Circular  A-76,  whose 
costing  methodology  included  forgone  federal  taxes  as  a  cost  element  for  in- 
house  producers. 

•  Fails  to  account  for  the  inherent  risk  of  cost  growth  among  public  producers. 
The  available  empirical  evidence  indicates  that,  for  competed  workloads, 
subsequent  cost  growth  depends  on  changes  in  the  size  and  scope  of  work, 
not  on  which  sector  wins.  The  DTM  approach  effectively  eliminates 
competition,  and  history  says  the  absence  of  competition  will  cause  cost  to 
increase  over  time. 

•  Overlooks  the  cumulative  cost  effect  of  multiple  in-sourcing  decisions.  Indirect 
costs  such  as  the  cost  of  payroll  processing  or  of  day-care  centers  do  not 
increase  as  the  result  of  any  single  in-sourcing  decision,  but  those  costs  will 
likely  rise  as  the  result  of  the  cumulative  effect  of  a  systematic  in-sourcing 
initiative. 

•  Overlooks  the  imputed  costs  of  insuring  and  indemnifying  in-house 
producers.  OMB  Circular  A-76  methodology  correctly  required  that  in-house 
producers  take  into  account  what  it  would  cost  if  they  were  required  to 
purchase  casualty  and  liability  insurance.  In  contrast,  the  DTM  recognizes  the 
costs  of  insurance  and  indemnification  to  private  producers,  but  there  is  no 
mechanism  in  the  DTM  that  attributes  such  costs  to  public  producers.5 

•  Fails  to  account  for  varying  workload  stability.  Some  tasks  require  a  rather 
constant  allocation  of  human  resources,  while  others  experience  high  levels 
of  volatility.  While  this  is  not  a  cost  factor  per  se,  the  flexibility  of  contractors 
can  provide  an  advantage  to  the  government  when  workload  is  variable,  and 
the  lack  of  flexibility  in  the  government  means  there  is  a  cost  to  maintaining 
an  unneeded  workforce  in  that  case. 

•  Should  require  a  detailed  scope  of  work  as  a  better  basis  for  cost  estimation. 
Such  a  detailed  scope  of  work  was  required  as  a  basis  for  cost  estimation  by 


5  Note  that  although  the  government  does  not  buy  insurance,  it  implicitly  insures  its  in-house 
producers.  The  cost  of  purchasing  insurance  reflects  the  expected  amount  of  these  costs. 
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the  A-76  process,  which  referred  to  that  scope  of  work  as  a  Performance 
Work  Statement.  Without  a  scope  of  work  that  accurately  lays  out  the 
requirements  of  the  task  to  be  performed,  it  is  impossible  to  ensure  that  the 
full  cost  of  performance  is  captured  in  any  cost  estimate. 

If  the  true  cost  of  public  performance  of  commercial  services  cannot  be  determined, 
any  budget-driven  decision  becomes  immediately  suspect,  whether  the  decision  is  to 
insource  work  currently  done  by  a  contractor  or  simply  to  change  the  size  of  a  specific  part 
of  the  government  workforce.  Such  a  situation  gives  rise  to  questions  like  “How  can  the  DoD 
claim  it  is  saving  40%,  or  25%,  or  any  specific  amount  via  in-sourcing  private-sector 
positions  if  it  doesn’t  know  how  much  the  newly  insourced  function  will  cost?”  and  “How  can 
the  Office  of  Management  and  Budget  approve  a  new  government  activity  if  it  does  not 
know  the  full  cost  impact  on  current  and  future  budgets?”  The  DoD  and  the  federal 
government  should  understand  the  full  budgetary  implications  of  every  personnel  decision 
so  that  it  can  properly  weigh  the  benefit  gained  (such  as  improving  in-house  capabilities) 
against  the  budgetary  impact. 

OMB  Circular  A-76 

OMB  Circular  A-76  provided  the  previous  cost  comparison  methodology  used  by  the 
DoD.  Given  the  flaws  of  the  DTM,  it  is  worth  considering  how  well  the  A-76  provides  a  basis 
for  addressing  those  flaws  and  performing  better  cost  estimates  of  government 
performance.  As  previously  discussed,  there  were  numerous  problems  with  the 
implementation  of  A-76  cost  competitions.  In  discussions  with  experts,  however,  there  was 
broad  agreement  that,  aside  from  the  two  specific  problems  discussed  below,  the  A-76 
costing  methodology  did  a  reasonably  good  job  of  accurately  capturing  the  major  cost 
elements  of  government  performance.6  Based  on  CSIS  analysis,  the  A-76  performs  better 
than  the  DTM  in  the  following  respects: 

•  provides  greater  specificity  on  major  cost  components, 

•  includes  the  cost  of  in-house  production  at  a  private  sector  rate  of  return  on 
new  investments, 

•  includes  forgone  federal  taxes  as  a  cost  element  for  in-house  producers, 

•  requires  that  in-house  producers  take  into  account  what  it  would  cost  if  they 
were  required  to  purchase  casualty  and  liability  insurance,  and 

•  requires  a  Performance  Work  Statement. 

Of  these,  the  most  important  is  the  fact  that  the  A-76  provides  far  greater  specificity 
on  major  cost  components,  thus  providing  better  guidance  for  cost  estimators  on  how  to 
compute  more  of  the  range  of  the  fully  burdened  cost.  In  contrast,  the  DTM  provides  only 
general  explanations  of  how  to  calculate  many  major  cost  elements  (aside  from  direct  labor 
costs). 

At  the  same  time,  A-76  still  exhibits  flaws  which  must  be  recognized  and  corrected. 

In  reviewing  the  literature  regarding  A-76,  the  majority  of  criticism  relates  to  the  competition 
process  itself  or  to  the  lack  of  follow-up  after  a  public-sector  victory  to  ensure  performance, 


6  It  should  be  noted  that,  while  the  experts  CSIS  spoke  to  for  this  study  agreed  that  the  A-76  cost¬ 
estimating  methodology  captured  most  of  the  major  cost  elements,  there  was  also  broad  agreement 
that  there  were  serious  weaknesses  in  the  quality  of  the  data  the  DoD  used  to  calculate  the  totals  for 
those  cost  elements. 
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rather  than  flaws  in  the  cost-estimation  methodology.  Two  major  criticisms  of  the  cost- 
estimation  system  itself  do  merit  discussion,  however: 

•  A-76  utilizes  a  blanket  12%  overhead  rate  for  all  government  functions.  In 
discussion  with  stakeholders  on  both  sides  of  the  sourcing  policy  debate,  as 
well  as  with  former  policy  makers  involved  in  A-76  drafting  and 
implementation,  there  was  agreement  that  the  12%  overhead  rate  lacks  any 
sound  methodological  basis,  and  that  it  was  wholly  inappropriate  to  have  one 
overhead  rate  to  cover  all  the  disparate  activities  performed  by  the 
government.  Industry  representatives  noted  that  private  sector  functions  with 
extremely  minimal  overhead  requirements  had  overhead  rates  two  to  three 
times  higher.  GAO  has  stated  that  the  12%  figure  came  not  as  the  result  of  a 
rigorous  study  of  government  overhead  costs,  but  as  a  compromise  between 
the  government  and  the  private  sector  (GAO,  1998,  p.  5). 

•  A-76  fails  to  account  sufficiently  for  the  true  cost  of  capital  on  the  public  side. 
A-76  is  better  in  this  respect  than  the  DTM,  which  includes  no  accounting  for 
cost  of  capital  while  forcing  contractors  to  account  for  it  in  their  pricing,  but 
further  research  is  needed  to  generate  a  methodology  for  fully  capturing 
public-sector  cost  of  capital. 

Current  Policy 

Within  the  DoD,  DoD  09-007  is  still  the  relevant  guidance  methodology  for  sourcing 
policy  and  cost  estimation.  In  discussions  with  policy  makers,  however,  CSIS  was  unable  to 
identify  a  single  office  or  function  that  was  utilizing  the  DTM  cost-estimation  methodology. 
Rather,  each  office  and  function  uses  whatever  cost-estimation  system  they  see  fit,  which 
has  led  to  situations  in  which  more  than  one  function  was  assuming  0%  overhead  rates  in 
calculating  its  own  costs.  The  DTM  was  supposed  to  have  been  replaced  with  a  more 
permanent  DoD  Instruction  by  September  201 1 ,  but  that  deadline  has  long  since  passed, 
and  the  revised  deadline  of  April  1, 2013,  was  extended  to  August  2013.  Indications  are  that 
the  DoD  Instruction  will  be  issued  no  later  than  May  2013.  In  addition,  the  model  that  the 
DoD  developed  to  aid  in  implementing  the  DTM  will  soon  be  available  for  use  throughout  the 
DoD.  Also,  the  GAO  is  preparing  a  report  for  Congress  on  DoD  guidance  and  compliance. 
The  release  date  for  this  GAO  report  is  not  yet  publicly  available.  CSIS  cannot  determine  at 
this  time  the  extent  to  which  the  DTM  shortcomings  cited  above  have  been  addressed  in  the 
Instruction  or  the  degree  to  which  the  GAO  will  agree  with  those  shortcomings.  CSIS  will 
update  this  section  in  the  final  report  as  further  information  becomes  available. 

Government-wide  action  on  workforce  costing  also  is  continuing.  On  March  1, 2013, 
the  Office  of  Federal  Procurement  Policy  (OFPP)  held  a  public  hearing  to  gather  information 
from  stakeholders  regarding  sourcing  and  cost-estimation  policy.  According  to  the  OFPP 
officials  at  the  event,  there  is  no  impending  rulemaking  from  the  OFPP  on  either  issue; 
rather,  the  OFPP  recognizes  that  these  are  issues  of  concern  to  various  stakeholders,  and 
they  are  trying  to  “get  smarter”  on  the  issue  in  advance  of  any  specific  policy  endeavor. 

Lessons  From  Business  Literature  on  Sourcing  Policy 

This  section  examines  some  of  the  relevant  literature  from  the  fields  of  economics 
and  business  management  for  insights  that  could  help  the  DoD  determine  which  services  to 
produce  in-house  and  which  to  purchase  under  contract  or  grant.  The  factors  that  private 
firms  consider  in  making  sourcing  decisions  have  withstood  the  test  of  competition  and  may 
provide  useful  guidance.  The  section  also  considers  findings  from  the  literature  on  public 
bureaucratic  behavior,  as  the  intrinsic  differences  between  governmental  and  private 
organizations  may  determine  the  ability  to  transfer  findings  from  the  private  sector 
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experience  to  the  government  realm.  Finally,  it  examines  empirical  studies  that — without  any 
theoretical  preconceptions — compare  the  costs  of  in-house  and  contractor  services  or 
examine  the  outcome  of  competitions  between  DoD  in-house  providers  and  private  sector 
contractors.7 

One  central  and  very  clear  finding  that  emerges  is  the  correlation  between 
competition  and  lower  costs.  For  many  DoD  commercial  activities,  the  cost  reduction 
associated  with  competition  is  on  the  order  of  20%  to  40%. 8  Both  the  business  and 
economics  literature  indicate  that  competition  provides  stronger  incentives  for  cost  reduction 
than  do  managerial  initiatives  that  monitor  performance  or  exhort  efficiency. 

The  Make-or-Buy  Decision  in  the  Private  Sector 

Sourcing  Decisions  From  an  Economist’s  Perspective 

Firms  are  sized  and  organized  to  maximize  the  value  of  output  less  both  the  costs  of 
production  and  the  costs  of  the  transactions  that  must  occur  between  the  different  players 
involved.9  The  literature  identifies  the  costs  of  transactions  and  of  information  as  important 
determinates  of  the  extent  to  which  firms  will  vertically  integrate — and  produce  their  own 
intermediate  goods  and  services — as  opposed  to  contracting  for  those  goods  and  services 
from  outside  producers  (Williamson  &  Winter,  1993).  Transactions  costs  occur  whenever 
goods  or  services  transfer  between  a  provider  (the  agent)  and  a  user  (the  principal).  The 
transactions  costs  associated  with  purchases  of  intermediate  goods  from  outside  suppliers 
include  the  costs  of  source  selection,  contract  management,  and  monitoring.  Those 
associated  with  in-house  production  include  the  costs  of  managing  labor  and  the  process  of 
obtaining  other  needed  inputs.  Transactions  between  principals  and  the  agents  on  whom 
they  rely  depend  on  governance  mechanisms — including  different  types  of  contracts  as  well 
as  incentives  and  performance  monitoring.  These  mechanisms  encourage  the  agents  to 
pursue  to  goals  of  the  principal. 

The  transactions  costs  associated  with  the  use  of  outside  providers  are  generally  low 
for  commercially  available  goods  that  can  be  purchased  off  the  shelf  and  for  generic 
commercial  services  that  can  be  performed  off-site — such  as  large-scale  data  entry. 
Accordingly,  these  are  the  kinds  of  goods  and  services  that  firms  often  choose  to  purchase 
rather  than  produce  internally.  The  transactions  costs  associated  with  using  outside 
producers  are  greater  if  the  outside  producer  must  invest  in  transaction-specific  assets  or 
skills  that  have  few  if  any  alternative  uses  (although  long-term  relationships  between  buyers 
and  sellers — which  in  effect  brings  the  workload  closer  to  in-house — can  help  to  alleviate 
this  problem). 

The  basic  findings  of  this  literature  are  that,  in  the  private  sector,  firms  find  that  it  can 
be  cost  effective  to  perform  work  in-house,  rather  than  by  contract,  if: 

•  Flexibility  is  required  to  meet  rapidly  changing  demands. 


7  The  focus  of  this  section  is  on  sourcing  decisions  for  activities  or  functions,  rather  than  on  in¬ 
sourcing  or  outsourcing  individual  positions  within  activities.  Because  changes  in  sourcing  typically 
change  the  quantity  of  labor  used,  a  comparison  of  costs  position  by  position  is  usually  not  relevant. 
The  special  situations  which  lead  the  DoD  to  contract  for  individual  positions — including  some  that 
could  be  inherently  governmental — are  set  aside  for  purposes  of  this  section. 

8  The  term  commercial  as  used  here  does  not  mean  a  good  or  product  that  is  readily  available  in  the 
commercial  sector;  it  means  only  that  the  activity  is  not  inherently  governmental  and  is  similar  to 
goods  or  services  that  are  available  in  the  private  sector. 

9  See  Simon  (1991).  For  a  nontechnical  summary  of  the  current  literature,  see  Williamson  &  Winter, 
1993. 
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•  The  quality  or  quantity  of  output  is  difficult  to  measure  objectively. 

•  In-house  workloads  are  large  and  steady  enough  to  provide  economies  of  scale. 

•  The  work  requires  highly  specialized  human  or  capital  assets. 

•  The  market  is  not  large  enough  to  support  competition  among  providers. 

•  The  work  requires  personnel  or  facilities  to  be  co-located  with  those  of  the  buyer 

(site  specificity).10 

These  factors  explain,  in  part,  why  it  can  be  more  difficult  to  contract  for  services 
than  for  goods.  Services  must  often  be  performed  at  the  buyer’s  site  to  meet  the  unique 
requirements  of  his  specific  production  process.  They  cannot  be  produced  in  advance  and 
then  sold  to  any  willing  buyer,  and  their  quality  and  quantity  may  not  lend  themselves  to  a 
physical  examination.* 11 

Sourcing  Decisions  From  a  Business  Management  Perspective 

The  business  management  literature  on  sourcing  decision,  although  consistent  with 
the  economics  literature,  identifies  some  additional  factors  that  influence  the  make-or-buy 
decisions  of  private  firms.12  Since  the  1990s,  this  literature  has  emphasized  that  a  firm’s 
competitive  advantage  often  rests  on  excellence  in  a  few  (perhaps  only  two  or  three)  “core 
competencies.”  Core  competencies  are  defined  as  “the  collective  learning  in  the 
organization,  especially  how  to  coordinate  diverse  production  skills  and  integrate  multiple 
streams  of  techniques”  (Prahalad  &  Hamel,  1990,  p.  82).  In  effective  organizations,  core 
competencies  are  closely  tied  to  the  values  of  an  organization  and  the  identities  of  its 
managers  and  employees — identities  which  those  values  can  help  shape.13  (This 
relationship  is  not  limited  to  business.  For  example,  in  the  military,  the  values  of  teamwork, 
loyalty,  and  honor  reinforce  the  core  competencies  of  combat  units.) 

The  time  and  effort  that  senior  managers  can  expend  on  non-core  activities  is 
limited.  The  support  functions  common  to  most  producers,  such  as  human  resource 
management  and  inventory  control,  although  essential  to  production,  are  often  not  among  a 
firm’s  core  competencies.  They  may  not  be  closely  monitored  or  controlled  by  the  most 
senior  managers  and — if  produced  in-house — are  not  directly  subject  to  market  forces.  In 
the  absence  of  direct  competition,  in-house  providers  may  fail  to  keep  up  with  the 
standards — for  quality  and  innovation  as  well  as  cost — that  outside  providers  must  meet. 
This  literature  suggests  that  non-core  activities  should  be  considered  for  outsourcing.  In 
addition  to  any  short-term  reduction  in  the  costs  of  obtaining  the  non-core  good  or  service, 
outsourcing  can 

•  free  up  management  to  focus  on  the  core  activities  that  drive  the  firm’s 
competitive  advantage, 

•  ensure  access  to  the  most  cutting  edge,  world-class  capabilities  that  could 
not  be  kept  in-house  cost  effectively, 

•  shift  risk  to  outside  providers,  and 


10  See  Pint  &  Baldwin  (1997)  and  Congressional  Budget  Office  (1995). 

11  Many  IT  services  may  lend  themselves  to  contracting  because  they  do  not  need  to  be  performed  on 
site. 

12  This  discussion  of  the  business  management  literature  draws  on  the  work  of  Pint  and  Baldwin 

mportance  of  identity  in  motivating  performance  has  recently  been  introduced  into  the 
economics  literature  (see  Akerlof  and  Kranton,  2005). 
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•  gain  control  over  what  could  otherwise  be  an  in-house  monopoly. 

The  business  literature  gives  less  emphasis  to  the  problems  that  firms  encounter  if 
their  outsourcing  decisions  are  poorly  conceived  or  implemented.  One  author  notes  that 
strategic  decisions  to  outsource  can  be  misapplied  by  line  managers  who  focus  narrowly  on 
short-run  cost  savings: 

While  outsourcing  may  seem  attractive  at  the  strategic  management  level, 
serious  pitfalls  are  often  encountered  as  the  strategy  is  pushed  downward 
into  operations.  At  the  operational  level,  the  strategic  intent  tends  to  be  lost ... 
implementation  is  in  the  hands  of  semiautonomous  teams  that  are  often 
tightly  focused  on  measureable  objectives — most  often,  cost  reduction. 
Outsourcing  at  the  operational  level  can  easily  lead  to  the  development  of 
dependencies  that  create  unforeseen  strategic  vulnerabilities.  (Insinga  & 
Werle,  2000,  p.  58) 

Although  the  DoD  has  adopted  some  of  the  language  of  the  business  literature,  it 
has  not  always  adopted  the  spirit.  For  example,  within  the  DoD,  the  need  for  direct 
command  and  control  is  often  cited  as  a  reason  why  specific  support  services  should  be 
kept  in-house.  This  is  consistent  with  military  culture  in  which  direct  authority  is  very 
powerful.  Yet  the  business  literature  indicates  that  senior  managers  in  the  private  sector  can 
often  extract  more  control  over  an  outside  provider  of  non-core  goods  or  services,  who 
operates  competitively,  than  they  do  over  an  in-house  monopolistic  provider  (a  provider  over 
whom  they  have,  at  least  nominally,  direct  authority;  Stiglitz,  1991,  pp.  15-24). 

Another  problem  in  applying  the  core  concept  is  that  it  is  difficult  to  distinguish  core 
from  non-core  competencies.  The  DoD’s  core  competencies  would  presumably  include  the 
application  of  military  force  in  support  of  national  security  objectives  as  well  as  other 
inherently  governmental  functions — including  the  control  of  public  funds  and  decision¬ 
making  that  commits  the  Department  to  an  action.  What  else  it  might  include  is  unclear.  For 
example,  the  DoD  uses  the  phrase  “core  workloads”  to  explain  why  some  depot 
maintenance  must  be  kept  in-house.  Yet  this  literature  suggests  that  specialized  workloads 
that  cannot  support  competition  or  the  need  to  maintain  the  expertise  to  be  a  successful 
buyer  might  be  better  justifications  for  some  in-house  capabilities. 

Making  Sourcing  Decisions  in  the  Private  Sector 

Both  the  economics  and  the  business  literature  indicate  that  workloads  can  exhibit 
characteristics  that  make  them  appropriate  for  in-house  production,  while  at  the  same  time, 
other  features  might  apparently  qualify  them  for  outsourcing.  Firms  must  consequently 
balance  the  different  characteristics  of  a  workload  when  making  sourcing  decisions.  This 
balancing  process  is  not  very  transparent.  The  2002  final  report  of  the  Commercial  Activities 
Panel,  a  group  chaired  by  David  Walker  of  the  GAO,  notes  that  private  sector  managers 
typically  review  the  merits  of  in-house  as  opposed  to  purchased  goods  and  services  at  a 
strategic  level  inside  the  organization  (CAP,  2002,  p.  108).  One  of  the  panel’s  witnesses 
indicated  that  cost  is  the  primary  consideration  in  only  a  third  of  private  sector  sourcing 
decisions. 

Direct  bidding  competitions  between  in-house  and  outside  providers  are  very  rare  in 
the  private  sector;  the  Commercial  Activities  Panel  was  unable  to  identify  any  such 
competition.  In  contrast,  it  is  not  uncommon  for  private  firms  to  maintain  both  in-house  and 
outside  providers  for  non-core  goods  or  services.  The  in-house  operations  provide  a  base  of 
expertise  for  evaluating  the  performance  of  the  specialist  providers  and,  if  the  market  is  thin, 
an  alternative  source  of  supply  and  a  form  of  implicit  competition  (Pint  &  Baldwin,  1997,  p. 

9). 
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The  Nature  of  Governmental  Organizations  and  the  Make-or-Buy  Decision 

Since  the  DoD  is  a  governmental  organization  and  not  a  private  firm  focused  on 
maximizing  profits  in  part  by  minimizing  costs,  its  decisions  on  outsourcing  will  be  driven  at 
least  in  part  by  factors  not  considered  in  the  business  and  economics  literature.  Indeed,  the 
business  literature  fails  to  explain  major  features  of  the  DoD’s  sourcing  policies.  For 
example,  a  key  factor  shaping  a  private  firm’s  decision  to  choose  in-house  production  for  a 
good  or  service  is  the  proximity  to  its  core  competencies  and  the  competitiveness,  or  lack 
thereof,  of  the  market.  Yet  a  recent  industrial  review  presented  to  the  Defense  Business 
Board  concluded  that  the  market  for  the  services  used  by  the  DoD  was  generally  highly 
competitive,  while  there  was  no  competition  for  the  production  of  aircraft  carriers,  tanks,  and 
ICBMs.  For  many  decades,  the  DoD  has  contracted  for  the  production  of  weapons  systems 
while  sourcing  policy  has  focused  on  contracting  for  services,  therefore  acting  in  direct 
contrast  to  practices  in  the  private  sector.  While  the  lack  of  competition  for  major  weapons 
systems  has  many  causes,  a  look  at  the  literature  dealing  with  governmental  agencies  and 
bureaucratic  behavior  provides  some  additional  insight  into  this  sourcing  practice. 

Constraints  and  Objectives  of  Public  Managers 

Both  the  classical  economics  literature  and  the  more  recent  work  on  the  behavior  of 
bureaucracies  suggest  that  public  producers  might,  in  theory,  be  both  less  anxious  and  less 
able  to  minimize  the  costs  of  production  than  their  private  counterparts. 

From  a  narrow  perspective,  the  only  intrinsic  difference  between  a  public  producer 
and  a  private  producer  is  that  one  is  owned  by  the  government  and  the  other  by  private 
individuals.  Accordingly,  the  economics  literature  asks  whether  a  government-owned  firm, 
operating  in  a  competitive  market  without  either  constraints  or  subsidies  (such  as  implicit 
loan  guarantees),  would  be  at  an  advantage  or  disadvantage  relative  to  a  private  firm.  The 
literature  concludes  that  public  production  is  at  a  disadvantage.  The  owners  of  the  private 
firm  can  more  readily  sell  their  firm  on  the  market  at  a  price  that  reflects  its  future  net 
earnings.  The  fact  that  the  value  of  the  investment  can  be  immediately  realized  gives  the 
private  firm  better  investment  incentives  (Alchian  &  Demsetz,  1972,  pp.  777-795). 

The  literature  on  government  agencies  and  public  bureaucracies  approaches  this 
question  from  a  much  broader  perspective.14  It  emphasizes  the  fact  that  government 
agencies  are  embedded  in  a  political  process.  A  federal  agency  serves  and  depends  on  the 
support  of  multiple  principals — including  public  interest  groups,  the  administration,  specific 
regulators,  the  Congress  as  a  whole,  and  specific  committees  within  the  Congress.  The 
agency  will  have  ambiguous  and  sometimes  conflicting  goals  as  the  result  of  compromises 
among  the  principals. 

Decisions  made  by  government  agencies  must  take  into  account  fairness  and 
accountability  in  addition  to  efficiency.15  Accountability  can  mean  making  decisions  in  a 
transparent  manner,  using  standard  operating  procedures,  even  if  allowing  managers 
greater  discretion  might  lead  to  more  efficient  outcomes.  It  can  also  mean  that  decisions  to 
commit  the  government  to  actions  must  be  taken  by  a  principal — an  elected  or  appointed 
official,  or  a  government  employee — whose  objectives  are  assumed  to  be  aligned  with  the 
public  interest,  rather  than  by  an  agent  seeking  merely  to  meet  the  terms  of  a  contract. 
Accountability  takes  on  great  importance  whenever  public  funds  are  being  expended.  Not 
only  must  the  process  for  expending  funds  be  followed,  but  the  agency  must  be  able  to 
demonstrate  this  clearly. 


14  Fora  clear  introduction,  see  Wilson  (1991). 

15  Efficiency  would  entail  an  output  produced  at  the  least  cost  as  well  as  a  budget  set  so  that  the 
benefits  to  society  from  additional  output  would  just  be  worth  the  additional  cost. 
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Fairness  is  of  particular  concern  in  the  area  of  labor  relations.  Here  the  government’s 
need  to  demonstrate  fairness  by  strict  adherence  to  standard  operating  procedures  is  further 
reinforced  by  the  desire  of  public  unions  and  employee  groups  to  use  similar  procedures  to 
protect  workers.  One  result  is  a  civil  service  system  with  its  strengths — an  ability  to 
withstand  demands  for  patronage — as  well  as  weaknesses  in  terms  of  the  limits  on 
managers’  discretion  to  hire,  fire,  promote,  and  pay.16  It  is  not  possible  for  an  agency  to 
satisfy  all  of  the  conflicting  objectives  of  its  multiple  principals.  Yet  as  long  as  agencies’ 
actions  are  seen  as  fair,  as  long  as  standard  procedures  are  followed,  its  actions  may  still  be 
accepted  and  criticism  deflected. 

In  addition  to  broad  public  goals  of  fairness,  accountability,  and  efficiency,  the 
literature  identifies  the  following  as  common  objectives  for  public  managers: 

•  providing  the  highest  quality  of  output; 

•  getting  the  highest  budget; 

•  obtaining  the  most  modern  technologies; 

•  being  fair  to  suppliers,  workers,  and  customers; 

•  offering  continuity  of  employment  to  workers;  and 

•  supporting  suppliers  who  may  be  small  or  disadvantaged.17 

In  some  cases,  these  reflect  the  goals  of  principals — either  what  they  desire  or  what 
they  perceive  to  be  in  the  public  interest.  In  other  cases,  they  reflect  the  goals  of  the 
agency’s  own  managers.  For  example,  in  addition  to  pursuing  their  principals’  goals, 
managers  may  seek  larger  budgets  or  staffs  as  signals  of  higher  prestige.18  Controlling  their 
own  levels  of  effort  is  also  a  concern.  The  difficulty  is  not  so  much  with  these  objectives 
(many  if  not  all  of  which  would  be  shared  by  private  managers)  but  that  public  managers 
may  be  less  constrained  by  market  forces  in  pursuing  them.  In  the  public  sector,  budget 
shortfalls  due  to  inefficiency  can  lead  to  an  increase  in  appropriated  resources.  The 
discretion  of  public  managers  to  pursue  their  own  objectives  is  particularly  great  when  they 
are  responsible  to  many  principals  with  conflicting  goals  (Dixit,  1997,  pp.  378-382). 

Principals  can  use  incentives  in  an  effort  to  align  their  agents’  actions  with  their 
goals.  Alternatively,  they  can  impose  external  constraints.  For  example,  in  the  past, 
Congress  has  placed  ceilings  on  DoD  civilian  employment  levels  and  on  the  size  of 
headquarters  activities.  Principals  can  also  set  performance  goals  (such  as  the  percentage 
of  commercial  activities  that  must  be  contracted  out  or  the  number  of  positions  that  must  be 
in-sourced)  and  monitor  performance.  Because  the  principals  do  not  have  access  to  much 
of  the  information  held  by  the  agents,  such  top  down  constraints  and  goals  will  often  appear 
(and  possibly  be)  arbitrary.  The  constraints  reduce  the  discretion  of  the  public  managers 
while  performance  rewards  can  distort  activities;  without  them,  however,  managers  may  not 
always  focus  on  the  goals  that  the  principals  feel  are  most  important.19 

Overall,  the  literature  on  the  behavior  of  public  bureaucracies  rejects  the  notion  that 
a  federal  agency  in  the  U.S.  could  mimic  a  competitive  firm — that  it  could  (or  should) 


16  See  Wilson  (1 991 ,  ch.  16  and  1 8)  for  a  discussion  of  how  rules  and  standard  operating  procedures 
protect  agencies  from  criticism. 

17  Many  of  these  goals  are  discussed  in  Wolf  (1988,  pp.  70-77). 

18  See  William  Niskanen’s  “budget-maximizing”  model. 

19  For  an  understanding  of  how  performance  measures  can  distort  incentives,  see  Heckman, 
Heinrich,  and  Smith  (1997). 
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completely  isolate  itself  from  concerns  about  fairness,  accountability,  and  public  welfare  that 
make  it  distinctly  governmental. 

Given  the  environment  in  which  government  agencies  work,  it  would  not  be 
surprising  if  the  DoD’s  sourcing  decisions  for  goods  and  services  simply  reflect  political 
realities.  A  reliance  on  in-house  production  for  services  may  reflect — in  addition  to  the  site- 
specific  and  perishable  nature  of  many  services — the  political  strength  of  the  civil  service 
and  the  fact  that  the  business  service  industry  and  its  labor  force  have  historically  been  less 
concentrated  and  powerful.  For  example,  from  an  efficiency  perspective,  there  is  no  reason 
for  14,000  civil  servants  to  be  employed  selling  groceries  to  military  personnel.  Although 
most  evidence  is  anecdotal,  one  study  of  the  outsourcing  decisions  of  3,000  county 
governments  between  1 987  and  1 992  found  quantitative  evidence  of  the  effect  of  politics — 
counties  with  highly  unionized  public  employees  chose  to  outsource  less  (Lopez-de-Silanes, 
Shliefer,  &  Vishnay,  1995). 

Is  It  Efficient  for  Public  Producers  to  Outsource  More  Than  Private  Producers? 

It  is  worth  asking  if  government  producers — to  the  extent  that  they  do  seek 
efficiency — would  find  outsourcing  even  more  attractive  than  do  private  producers.  In  the 
case  of  labor-intensive  services,  limitations  in  the  ability  of  federal  managers  to  hire,  fire, 
promote,  and  pay  would — even  by  itself — seem  to  dictate  this.  Two  factors,  however,  may  at 
least  partially  offset  these  motivations. 

One  is  the  need  for  the  government,  operating  with  public  funds  in  the  public  interest, 
to  keep  fraud  and  conflicts  of  interest  to  a  minimum.  A  private  firm  might,  in  some  situations, 
outsource  some  of  its  financial  management  or  decision-making  and  treat  any  loss  due  to 
contractor  fraud  or  conflicts  of  interest  as  a  simple  cost  of  doing  business.  For  a  government 
agency,  however,  such  losses  are  tied  to  functions  that  would  be  considered  inherently 
governmental — something  for  which  the  agency  must  be  directly  accountable  to  the  public. 

A  second  reason  is  that  the  same  factors  that  make  the  government  less  efficient  as 
a  producer  of  goods  and  services  also  make  it  less  efficient  as  a  buyer.20  The  literature 
relating  to  the  need  for  reform  of  the  civil  service  system  is  matched  by  that  citing  the  need 
for  acquisition  reform.  The  need  to  demonstrate  fairness  and  transparency,  for  example,  can 
make  it  hard  for  contracts  to  be  awarded  to  any  but  the  lowest  cost  bidder,  irrespective  of 
more  subjective  concerns  about  performance.  The  balancing  of  competing  objectives  that 
private  sector  managers  appear  to  use  in  making  sourcing  decisions,  however  effective  over 
the  long  run,  would  not  readily  stand  up  to  scrutiny  by  the  GAO  or  an  Inspector  General 
concerned  with  transparency  and  accountability. 

Public  and  Private  Production:  The  Evidence  From  Outside  the  DoD 

DoD  outsourcing  decisions  would — in  theory — be  simplified  if  there  was  strong 
evidence  that  government  production  under  competition  was,  empirically  as  well  as 
conceptually,  more  costly  than  private  production.  Some  commercial  activities  would  be  kept 
in-house  because  of  acknowledged  non-cost  benefits  of  in-house,  rather  than  private 
production.  How  many  and  which  ones  would  remain  a  source  of  controversy,  but  the 
remaining  commercial  workloads — current  as  well  as  future  ones — could  be  shifted  to  the 
private  sector  without  the  need  for  questionable  cost  analyses  or  disruptive  direct 
competitions. 

Economists  may  be  willing  to  conclude  on  conceptual  grounds  that — in  markets  with 
strong  competition  and  no  market  failure — the  public  sector  has  at  least  no  intrinsic 


20  For  a  discussion  that  links  government  problems  as  a  buyer  with  the  nature  of  public 
bureaucracies,  see  Kelman  (1990). 
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advantage  over  private  production.  Yet  the  DoD,  faced  with  the  concerns  of  public 
employees  and  the  imperfections  of  real  (and  often  defense-specific),  as  opposed  to 
idealized,  markets,  might  need  somewhat  more  concrete  arguments  to  make  the  case  for 
advantages  over  the  private  sector  for  commercial-type  activities.  What  does  the  empirical 
evidence,  including  that  from  the  public-private  competitions  conducted  within  the  DoD, 
indicate  about  the  relative  costs  of  public  and  private  production? 

In  developed  economies,  public  and  private  producers  are  not  often  found  side  by 
side  in  competitive  markets,  and  analytical  evidence  about  the  relative  performance  of  public 
and  private  enterprises  under  competition  is  limited.  Nonetheless,  there  have  been 
hundreds  of  studies  comparing  public  and  private  productions,  as  well  as  numerous  reviews 
of  that  literature.21 

The  findings  of  studies  often  depend  on  the  type  of  data  used.  Comparisons  between 
the  performance  of  public  and  private  enterprises  in  Europe  have  focused  on  industries  such 
as  steel  or  transportation  in  which  economies  of  scale  or  public  regulation  limit  competition. 
Many  of  these  studies  have  found  that  public  provision  is  less  costly.  In  contrast,  studies  that 
focus  on  more  competitive  activities — such  as  waste  collection,  street  cleaning,  or  routine 
building  maintenance  — that  can  either  be  performed  or  purchased  by  local  governments, 
generally  find  that  private  provision  is  less  costly  (Borcherding,  Pommerehne,  &  Schneider, 
1982,  pp.  127-156).  In  these  studies,  however,  the  cost  differential — which  is  often  on  the 
order  of  20%  to  30% — often  reflects  not  only  any  intrinsic  advantage  of  private  production 
but  also  the  effects  of  introducing  competition. 

Overall,  the  studies  that  most  strongly  assert  the  efficiency  of  private  over  public 
production  are  often  those  that  rely  on  the  weakest  evidence,  and  some  careful  reviewers 
doubt  that  there  is  credible  evidence  that  private  production  has  any  intrinsic  advantage  in 
relation  to  public  production  (Stiglitz,  1991,  pp.  15-24). 

Overall,  this  literature  leads  to  the  following  conclusions: 

•  Public  production  might  be  less  efficient  than  private  production. 

•  If  public  production  is  less  efficient,  the  difference  may  be  insignificant. 

•  Competition  seems  to  drive  efficiency  more  than  does  the  form  of  ownership. 

How  is  this  empirical  literature  to  be  reconciled  with  what  is  known  about 
bureaucratic  behavior  and  the  costs  that  government  agencies  incur  in  managing  labor  and 
other  resources  so  as  to  both  demonstrate  and  provide  fairness  and  accountability? 

One  answer  is  that  any  public  enterprise  that  survives  in  competition  with  the  private 
sector  on  a  level  playing  field  is  only  public  in  the  sense  that  it  is  a  business  owned  by  the 
government.  If  the  playing  field  is  truly  level,  it  cannot  rely  on  public  funds  or  the  political 
process  for  its  survival  and  is  thus  by  definition  less  of  a  government  agency  in  the 
bureaucratic  sense.  Some  authors  suggest  that,  under  these  peculiar  circumstances,  its 
form  of  ownership  has,  in  practice,  changed  from  “public”  to  “private”  (Boardman  &  Vining, 
1992,  pp.  205-239).  The  fact  that  the  residual  value  of  the  enterprise  accrues  to  the 
government  rather  than  to  private  individuals  may  not  greatly  affect  its  efficiency. 

Another  answer  is  that  the  playing  field  may  be  tilted  by  hidden  subsidies,  such  as 
forgone  taxes  and  import  duties.  Some  authors  suggest  that  the  apparent  success  of 
government  enterprises  in  capital  intensive  industries  is  due  in  part  to  a  hidden  capital 
subsidy  (Ayab  &  Hegstand,  1987,  pp.  79-101).  The  government’s  borrowing  rate — which 


See,  for  example,  Tighe  et  al.  (1997). 
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reflects  its  ability  to  raise  taxes  to  cover  its  borrowing — will  typically  be  lower  than  that  faced 
by  a  private  firm.  Yet  capital  devoted  to  a  risky  commercial  activity  is  not,  in  any  real 
economic  sense,  less  costly  if  it  is  undertaken  by  the  government  rather  than  a  private 
entity. 

Each  of  these  issues  offers  the  potential  for  additional  research.  However,  whether 
that  research  addresses  the  specific  issues  of  the  level  playing  field  or  the  broader  question 
of  capital  budgeting  for  asset  amortization  and  depreciation,  it  will  be  years  before  the 
results  are  available.  In  the  meantime,  public  policy  needs  to  use  available  data  to  make  the 
best  decisions  available. 

Towards  a  More  Methodologically  Sound  Sourcing  Policy 

Regardless  of  the  future  of  the  DoD’s  in-sourcing  initiatives,  it  seems  likely  that 
sourcing  policy  will  be  a  continuing  source  of  debate  and  concern  to  policy  makers  going 
forward.  In  a  time  of  budgetary  uncertainty  and  decline,  stakeholders  on  all  sides  of  the 
issue  will  continue  to  press  their  cases  for  how  the  DoD  can  best  utilize  resources  to 
execute  the  missions  it  is  tasked  to  perform.  CSIS  believes  that  the  only  way  for  the  DoD 
and  the  OMB  to  make  meaningful  progress  on  these  issues  is  to  develop  methodologies 
based  on  the  best  and  most  complete  data  available.  As  discussed  earlier  in  this  paper,  this 
approach  will  have  benefits  in  any  decision  the  DoD  makes  that  has  budgetary  implications. 
Policy  makers  should  always  have  the  most  accurate  picture  available  of  the  true,  fully 
burdened  cost  implications  of  the  choices  before  them. 

Unfortunately,  the  literature  on  how  the  private  sector  approaches  sourcing  decisions 
does  not  appear  to  offer  many  lessons  for  the  public  sector.  The  way  the  private  sector 
defines  core  competencies  and  focuses  on  keeping  those  in-house  may  provide  some 
useful  lessons  learned  as  the  OFPP  continues  to  refine  its  guidance  on  what  functions  or 
positions  qualify  as  inherently  governmental  or  closely  associated.  But  overall,  there  are  too 
many  differences  between  the  way  decisions  are  made  and  how  various  costs  and  benefits 
are  weighted  to  allow  for  useful  comparisons  between  how  public  and  private  entities 
approach  sourcing  decisions. 

In  the  final  stages  of  this  research  effort,  CSIS  will  be  expanding  upon  its  previous 
work  on  the  subject  to  support  the  development  of  a  repeatable,  verifiable,  data-driven 
approach  calculating  the  cost  of  government  performance.  The  CSIS  approach  focuses  on 
line-item  specificity  for  cost  elements,  tied  to  a  detailed  Statement  of  Work  based  on  the 
elements  in  the  following  taxonomy  (see  Figure  1).  CSIS  has  verified  that  data  exist  within 
DoD  systems  to  support  at  a  minimum  the  ability  to  calculate  a  range  of  cost  estimates  for 
each  of  these  elements.  Using  the  data  for  cost  estimating  and  decision-making  will  hasten 
the  improvement  of  both  data  and  cost-estimation  methodologies. 
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The  CSIS  Public  Cost  Estimation  Taxonomy 
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Figure  1.  The  CSIS  Public  Cost  Estimation  Taxonomy 

(Berteau  et  al.,  2011,  p.  16) 


To  build  upon  this  taxonomy,  CSIS  plans  to  incorporate  OMB’s  Object  Class  Codes 
(OCCs)  to  provide  even  greater  specificity  of  cost  elements.  OCCs  are  used  by  agencies, 
including  the  DoD,  for  internal  financial  tracking  and  by  congressional  staff  for 
appropriations,  and  CSIS  believes  that  tying  a  cost-estimating  methodology  to  this  widely 
used  and  well-understood  cost  classification  system  will  provide  a  basis  for  a  realistic, 
implementable  methodology  for  capturing  the  true,  fully  burdened  cost  of  government 
performance.  By  relying  on  existing  data  to  the  maximum  extent  possible,  the  DoD  can  find 
it  easier  to  calculate  better  cost  estimates  and  to  use  those  estimates  in  sourcing  and  other 
budget  decisions. 
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Abstract 

In  Department  of  Defense  (DoD)  acquisition  contracts  there  are  often  concerns  of  security 
and  competitive  advantage  making  it  difficult  to  find  comparable  performance  data  that  may 
be  useful  in  evaluating  contractor  proposals.  In  order  for  programs  to  make  such  comparative 
evaluations,  a  should-cost  analysis  may  be  conducted.  This  analysis  can  be  compared  to  a 
benchmarking  process  provided  that  a  benchmark  database  is  available.  Parametric 
estimation  tools  provide  this  type  of  data. 

This  paper  shows  how  SEER-SEM  was  applied  as  part  of  the  should-cost  effort  on  the  F-22 
program.  The  Office  of  the  Secretary  of  Defense  recognized  the  resulting  $32  million  savings 
in  the  presentation  on  Better  Buying  Power  II. 

Introduction 

June  28,  2010,  Under  Secretary  Ashton  Carter  issued  the  Better  Buying  Power 
memorandum  (Carter,  2010)  suggesting  seven  (7)  focus  topics.  “Should-cost  analysis” 
addresses  several  of  the  focus  areas  but  most  clearly  the  one  Secretary  Gates  labeled 
“Incentivize  Productivity  and  Innovation  in  Industry  and  Government.”  The  Department  of 
Defense  (DoD)  has  significant  history  with  should-cost  analyses.  A  RAND  study  (Boito, 
2012),  examined  this  history  from  the  1970s  to  today.  The  RAND  study  finds  support  for  this 
analysis  in  the  Federal  Acquisition  Regulations  (FAR)  as  follows: 

Should-cost  analysis  as  described  in  the  FAR  is  a  specialized  form  of  cost 
analysis,  used  to  support  contract  negotiations,  that  is  characterized  by  a 
focus  on  the  elimination  of  contractor  inefficiencies.  It  is  significant  that  the 
guidance  for  should  cost  analysis  is  found  in  the  federal  regulation  for  the 
contracting  function,  because  contracting  is  the  process  by  which  the 
government  specifies  what  it  wants  to  buy  and  at  what  price.  (Boito,  2012,  p. 
41) 
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In  this  study,  RAND  observes  that  a  should-cost  analysis  requires  participation  of 
both  contractors  and  government  personnel.  Successful  negotiation  can  only  be  achieved 
when  the  contractor  agrees  to  the  objectivity  of  government  observations  and  the  contractor 
believes  it  can  eliminate  the  inefficiency.  The  negotiation  task  is  often  difficult  because  the 
government  is  frequently  in  a  position  of  having  a  single  source  supplier.  The  single-source 
situation  may  make  it  difficult  for  the  government  to  persuade  the  contractor  to  participate 
openly  in  the  should-cost  analysis.  Any  lack  of  openness  or  access  to  data  will  limit  the 
government’s  ability  to  identify  the  inefficiencies. 

A  major  challenge  in  conducting  a  should-cost  analysis  is  the  skill  required  of  the 
analysts.  The  team  doing  the  analysis  must  encompass  skills  in  pricing,  contracting, 
program  management,  and  subject  matter  expertise  in  areas  relevant  to  the  program  (Boito, 
2012,  p.  x).  This  team  must  have  both  depth  of  knowledge  in  the  focus  disciplines  and 
breadth  of  experience  across  programs  and  industry.  Finally,  they  must  be  able  to  apply 
these  skills  to  present  an  objective  set  of  recommendations  accessible  to  both  program 
management  and  contractor. 

The  Software  Engineering  Institute  (SEI)  has  participated  in  some  should-cost 
analyses  using  parametric  software  cost  estimation  tools.  This  paper  describes  the 
methodology  and  some  results.  The  following  section  describes  the  methodology.  Then 
next  section  discusses  an  example  application  and  results  synthesized  from  multiple  cases. 
The  final  section  provides  lessons  learned  and  ideas  for  future  improvements. 

SEI  Should-Cost  Methodology 

The  DoD  may  have  gotten  an  early  start  on  everyone  with  “should-cost  analysis,”  but 
the  commercial  world  has  pursued  the  topic  extensively  under  the  label  of  “benchmarking.” 
An  early  book  on  the  subject  is  Benchmarking:  The  Search  for  Industry  Best  Practices  That 
Lead  to  Superior  Performance  by  R.C.  Camp  (Camp,  1989).  Just  a  year  later,  James 
Womack,  Daniel  T.  Jones,  and  Daniel  Roos  (1990)  described  Toyota’s  use  of  benchmarking 
in  The  Machine  That  Changed  the  World.  In  the  1990s,  corporate  benchmarking  was  a 
popular  consulting  business. 

The  SEI  should-cost  work  stemmed  directly  from  SEI  experience  with  benchmark 
databases  in  the  form  of  parametric  cost  estimation  tools.  Using  the  parametric  estimation 
tools  is  not  quite  the  same  approach  as  traditional  benchmarking,  but  the  cost  of  this 
approach  is  modest  and  works  well  considering  the  resistance  to  traditional  benchmarking  in 
the  DoD  acquisition  context. 

Five  steps  are  required  to  prepare  a  should-cost  proposal  using  parametric 


estimation  tools. 

Step  1: 

Develop  a  detailed  understanding  of  the  proposer’s  estimate.  Include 
product  scope,  architecture,  and  methods  of  development  by 
reviewing  the  proposal  and  proposer’s  basis  of  estimate. 

Step  2: 

Use  a  parametric  estimation  tool  to  develop  an  estimate  that  matches 
the  proposer’s  estimate  as  closely  as  possible.  Estimates  of  size 
must  match  exactly. 

Step  3: 

Perform  a  sensitivity  analysis  to  identify  the  productivity  factors  having 
the  greatest  effects  on  program  performance. 

Step  4: 

Prepare  an  alternative  estimate  with  the  adjusted  parameters. 

Develop  a  briefing  demonstrating  the  changed  parameters  and  new 
estimate. 
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Step  5:  Conduct  a  workshop  to  help  the  contractor  plan  potential  performance 
improvements. 

Step  1:  Develop  a  Detailed  Understanding  of  the  Proposer’s  Estimate 

This  step  will  require  access  to  many  details  of  the  contractor’s  basis  of  estimate  and 
some  interviews  with  the  contractor’s  staff.  This  step  requires  access  to  the  program 
management  and  engineering  staff  who  provided  the  size,  product  complexity,  and  project 
environment  factors  used  for  the  estimate.  Usually,  the  interviews  will  require  a  full  day  and 
may  require  an  additional  phone  call  to  understand  the  contractor’s  meaning  and  intent  for 
some  data.  Analyzing  the  basis  of  estimate  may  require  as  much  as  five  to  seven  days  in 
total.  Understanding  the  scope  of  work  and  complexity  of  the  proposed  product  is  not  easy 
since  the  WBS  (e.g.,  task  sheets)  structure  of  the  proposal  may  cause  parts  of  the  estimate 
to  be  represented  in  several  different  sections. 

•  Begin  preparation  by  reviewing  product  requirements,  including  proposed 
product  architecture.  Identification  of  complexity  factors  such  as  aggressive 
key  performance  measures,  safety,  interfaces,  and  others  will  be  essential  to 
preparing  the  estimate. 

•  Provide  the  contractor  with  requirements  for  data  and  interviews. 

With  the  contractor,  complete  the  following: 

•  Review  analogies  used  for  developing  the  size  estimate.  Did  setting  the  size 
follow  a  standard  procedure  used  previously  by  the  company?  Is  there  any 
reason  the  size  would  have  been  adjusted  to  meet  a  target  price?  Use  these 
factors  to  set  a  potential  range  for  the  size  estimate. 

•  Check  the  scope  definition  to  see  which  components  and  work  products  will 
be  delivered  and  to  whom  they  will  be  delivered.  Count  every  delivery  outside 
the  development  team  (e.g.,  product  certification  and  public  demonstration). 

•  Check  the  domain  definition  and  whether  the  product  is  considered  to  be  new 
or  a  modification  and  enhancement. 

•  Identify  the  collection  of  task  sheets  representing  the  WBS  that  will  be  utilized 
by  the  estimation  tool.  Sum  up  the  efforts  on  these  task  sheets  that 
correspond  to  the  estimation  tool  outputs. 

•  Review  the  definition  and  computation  of  application  complexity.  Specifically 
look  for  performance  criteria  and  quality  attributes  that  may  represent  specific 
baseline  attributes  in  the  estimation  tool  knowledge  base.  This  step  is 
important  because  there  may  be  inconsistencies  between  the  proposer’s  use 
of  terminology  and  the  tool’s  knowledge  base  use  of  the  same  terms.  For 
instance,  some  performance  requirements  might  use  the  phrase  “real  time”  to 
mean  “very  fast”  where  the  normal  interpretation  is  “deadline  driven.” 

•  Review  “Manager’s  Checklist  for  Validating  Software  Cost  and  Schedule 
Estimates”  (Park,  1994)  to  confirm  satisfaction  with  the  contractor’s 
estimation  process  and  resulting  basis  of  estimate. 

•  Document  the  size  estimate  and  the  knowledge  base  factors  to  be  applied  for 
each  component  that  will  be  estimated.  The  size  values  should  be  the  current 
baseline  product,  proposed  reuse,  modification,  and  new  development.  Use 
of  proxy  measures  such  as  ESLOC  will  add  uncertainty  to  the  estimate. 
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At  the  completion  of  this  step,  you  should  be  ready  to  supply  the  parametric  inputs  in 
the  next  step. 

Step  2:  Match  Proposer’s  Estimate 

The  purpose  of  this  step  is  to  use  the  parametric  tool  to  produce  an  estimate  that 
matches  the  contractor’s  estimate  as  closely  as  possible.  The  estimates  should  match, 
within  a  small  difference  on  size,  effort,  schedule,  and  defects.  Many  different  parameters 
must  be  tested  to  achieve  a  satisfactory  result. 

Perform  the  following  activities  during  this  step: 

•  Clearly  identify  as  much  of  product  context  as  the  tool  allows.  Most  tools 
allow  specification  of  product  domain  (e.g.,  avionics),  development 
methodology,  and  development  language. 

•  Begin  by  entering  base,  new,  modified,  and  deleted  size  estimates.  ESLOC 
can  be  used  as  a  last  resort,  but  this  increases  the  uncertainty  in  the 
estimate.  It  is  not  possible  to  use  an  ESLOC  value  to  back  out  the  base,  new, 
modified,  and  deleted  values. 

•  Record  additional  estimation  tool  parameter  values  such  as 

o  available  tools  and  platforms, 

o  experience  of  team  members  in  both  development  and  architecture, 
o  organizational  process  maturity, 
o  quality  assurance  and  testing,  and 

o  factors  affecting  team  performance,  such  as  cohesion  and 
geographical  proximity. 

Detailed  familiarity  with  the  parametric  tool  is  required  for  this  step.  DoD 
contractors  are  and  will  claim  to  be  high-caliber  development  organizations. 
Interviews  are  a  good  mechanism  for  obtaining  the  parameter  values,  but 
experience  and  judgment  are  necessary  for  trustworthy  results. 

•  Modify  the  parameter  values  of  the  baseline  to  match  the  contractor  estimate. 
This  step  may  be  difficult  and  tedious.  Even  a  fairly  simple  tool  like  COCOMO 
II  has  22  factors  affecting  productivity  plus  various  sizing  factors.  Once  the 
initial  estimate  is  prepared  with  contractor  sizing  and  product  domain 
information,  it  is  time  to  match  the  contractor  estimate  by  adjusting  quality 
and  productivity  parameters. 

•  Save  the  matched  estimate  as  a  baseline. 

If  no  reasonable  match  can  be  made,  then  it  is  time  to  re-check  the  Park  (1994)  checklist 
and  re-interview  the  contractor.  Most  likely,  there  is  a  misinterpretation  of  some  size 
measure,  knowledge  base  parameter,  or  performance  parameter.  It  is  also  possible  that  the 
contractor’s  WBS  has  been  misinterpreted. 

Step  3:  Perform  Sensitivity  Analysis 

The  sensitivity  analysis  is  necessary  in  order  to  make  concrete  suggestions  about 
productivity  improvements.  Productivity  parameters  will  include  such  factors  as  team 
cohesion,  developer  experience,  project  environment,  and  process  maturity.  Product  quality 
parameters  will  address  questions  about  the  target  environment,  testing,  and  stability  of  the 
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specification.  Parameters  affecting  product  quality  should  generally  be  excluded  from  the 
sensitivity  analysis  unless  some  error  has  been  identified  in  the  proposal. 

•  If  the  tool  provides  a  sensitivity  analysis,  then  use  the  suggested  top  10 
parameters  for  improvement  potential.  If  the  tool  lacks  this  capability,  it  may 
be  necessary  to  apply  brute  force  or  Monte  Carlo  methods  to  determine  the 
parameter  sensitivity. 

•  List  the  parameters  to  be  tested  for  alternative  estimates. 

Step  4:  Prepare  Alternative  Estimates 

•  Re-run  estimates  with  the  identified  performance  criteria  set  to  revised 
values.  The  revised  values  are  selected  from  benchmark  data.  These  values 
may  be  taken  from  the  best  projects  in  the  tool  vendor’s  database  or  another 
source. 

•  Document  the  alternative  schedule,  effort,  and  defects  along  with  the  revised 
resource  allocation  (how  much  effort  is  suggested  for  top  few  roles). 

•  Save  the  new  baselines  with  identification. 

•  Document  the  changes  to  the  affected  parameters. 

•  Document  the  differences  from  the  contractor’s  baseline  in  schedule,  effort, 
defects,  and  cost. 

•  Run  a  second  sensitivity  analysis.  If  the  sensitivity  analysis  suggests 
significant  additional  improvements  are  possible,  then  repeat  this  step  and 
develop  a  second  should-cost  estimate  and  proposal. 

Summarize  the  results  in  a  briefing  making  comparisons  of  estimated  results  and 
alternative  parameter  values.  Associated  with  each  alternative  should  be  a  discussion  of  the 
rationale  for  the  potential  improvements  and  how  they  might  be  achieved.  If  more  than  one 
estimate  will  be  presented,  then  be  prepared  to  discuss  the  relative  improvement  achieved 
by  each. 

Step  5:  Workshop 

The  workshop  begins  with  a  presentation  of  the  analytical  results  and  concludes  with 
some  recommendations  for  action.  A  workshop  is  necessary  as  the  contractor  must  agree  to 
planning  and  resourcing  to  make  changes. 

•  Display  the  baseline  estimate  beginning  with  the  usual  values:  size,  effort, 
schedule,  and  defects. 

•  Show  the  sensitivity  analysis  used  to  arrive  at  the  new  estimation  parameter 
values. 

•  Provide  the  actual  list  of  parameter  values  applied  for  the  new  estimate. 

•  Display  the  revised  estimate  showing  the  comparison  of  the  values  to  the 
baseline. 

•  Provide  comparisons  and  explanations  of  initial  and  revised  parameter 
values. 

•  Allow  contractor  evaluation  of  potential  for  change. 

•  Achieve  agreement  on  action  items  to  resource  changes. 
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Results 

The  SEI  participated  with  AT  Kearney  (ATK)  in  a  should-cost  analysis  for  the  F-22 
3.2a  contract.  The  SEI  used  the  method  described  here  and  ATK  applied  bottom-up 
analysis.  Both  approaches  led  to  very  similar  cost  savings,  which  gave  the  resulting 
recommendations  very  strong  weight.  As  a  result  of  this  should-cost  effort,  the  program 
office  was  able  to  negotiate  a  15%  reduction,  $32  million  cost  savings.  These  results  were 
reported  in  a  recent  OSD  (2012)  publication  Better  Buying  Power  II. 

There  were  several  lessons  learned  during  this  effort.  Many  of  the  lessons 
correspond  to  the  recommendations  in  the  aforementioned  RAND  report. 

1 .  A  dedicated  independent  team  is  needed.  This  team  was  focused  on  the 
should-cost  effort  and  not  distracted  by  contracting  and  immediate  technical 
problems. 

2.  Use  of  multiple  methods  for  should-cost  has  value  to  program.  The  methods 
used  by  ATK  and  SEI  were  independent  and  different.  The  results  were 
similar  and  carried  a  great  deal  of  weight  in  negotiations  because  of  the 
independence. 

3.  A  contractor’s  estimation  procedure  based  solely  on  historical  data  is 
insufficient.  Such  contractors’  estimates  may  be  defensible  but  miss  the 
opportunity  for  benchmarking  against  competition  and  industry-wide 
comparisons.  Should-cost  is  a  method  that  requires  available  benchmarks  for 
both  cost  and  quality  and  specifically  identifies  the  driving  factors  behind  cost 
and  quality. 

4.  The  contractors’  usage  of  estimation  tools  must  be  examined  carefully. 
Contractors  may  change  the  cost  estimation  tool’s  baseline  data  in  order  to 
match  contractor  performance  history.  This  approach  can  compromise  the 
ability  to  use  the  parametric  model  as  a  baseline.  Using  the  parametric  model 
as  a  benchmark  required  significant  analysis  to  arrive  at  a  baseline  value  that 
matched  the  contractor’s.  Contractors  had  misinterpreted  some  input 
productivity  factors  and  adjusted  the  output  calculations  instead. 

5.  Not  all  parameters  are  easy  to  identify.  For  example,  SEER  makes  use  of  a 
parameter  that  can  be  used  to  account  for  independent  development  teams 
when  size  has  not  been  partitioned  to  the  component  level  in  the  estimate. 
Partitioning  the  work  allows  for  a  more  aggressive  schedule  estimate  since 
teams  are  able  to  operate  independently  until  integration  testing.  This  may  be 
difficult  to  detect  from  the  available  documentation. 

6.  Consider  the  effects  of  adding  automation  or  tooling  to  testing  and  other 
process  changes.  Cost  savings  are  often  made  possible  by  making  process 
changes;  however,  process  changes  can  take  time  to  execute.  Some  savings 
that  were  suggested  in  the  F-22  analysis  were  not  achievable  within  the  time 
horizon  of  the  3.2a  effort.  Recommendations  will  be  accepted  or  rejected  as 
part  of  the  negotiation  process. 

There  were  a  number  of  reasons  to  consider  the  F-22  analysis  a  success.  The 
government  certainly  was  happy  to  negotiate  a  better  price.  Even  though  some  of  the  work 
between  analysts  and  contractors  was  contentious,  the  contractors  were  able  to  agree  to  a 
number  of  suggested  improvements.  An  additional  should-cost  analysis  was  also  conducted 
for  the  next  contract  block.  The  second  time  through  there  was  already  evidence  of 
improved  performance  and  much  less  contention  during  the  analysis. 
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It  will  be  a  while  before  the  final  numbers  are  available  from  the  F-22  modernization 
work.  Hopefully,  that  will  also  be  a  success  story. 
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Abstract 

The  2009  Weapon  System  Acquisition  Reform  Act  (WSARA)  requires  the  DoD’s  Office  of 
Cost  Assessment  and  Program  Evaluation  (CAPE)  to  “periodically  assess  and  update  the 
cost  (or  inflation)  indexes  used  by  the  Department  to  ensure  that  such  indexes  have  a  sound 
basis  and  meet  the  Department’s  needs  for  realistic  cost  estimation.”  The  objective  of  this 
paper  is  to  provide  CAPE  with  a  factual  and  analytical  basis  for  responding  to  this  provision  of 
WSARA. 

The  paper  starts  with  a  discussion  of  the  rationale  for  using  inflation  indexes  in  general,  in  the 
government  as  a  whole,  and  in  the  DoD.  It  then  identifies  the  regulatory  and  statutory 
provisions  that  support  the  issuance  of  inflation  guidance  by  the  Under  Secretary  of  Defense 
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(Comptroller;  USD[C]).  Next,  it  describes  how  this  guidance  is  applied  by  describing  the  key 
features  of  the  processes  used  in  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the 
Services  to  adjust  for  inflation  in  estimating  the  costs  of  and  budgeting  for  major  systems.  It 
evaluates  the  appropriateness  of  using  the  inflation  indices  provided  by  the  USD(C).  Finally,  it 
compares  the  Comptroller’s  rates  with  some  alternatives  and  considers  whether  modifications 
to  current  practices  might  better  meet  the  DoD’s  needs  for  realistic  cost  estimation. 

Introduction 

The  2009  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  requires  the  DoD, 
Office  of  Cost  Assessment  and  Program  Evaluation  (CAPE)  to  “periodically  assess  and 
update  the  cost  (or  inflation)  indexes  used  by  the  Department  to  ensure  that  such  indexes 
have  a  sound  basis  and  meet  the  Department’s  needs  for  realistic  cost  estimation.”  The 
objective  of  this  paper  is  to  provide  CAPE  with  a  factual  and  analytical  basis  for  responding 
to  this  provision  of  WSARA.  Because  WSARA  is  concerned  with  the  cost  of  major  systems, 
much  of  our  attention  is  given  to  the  treatment  of  inflation  by  Major  Defense  Acquisition 
Programs  (MDAPs). 

In  the  next  section,  we  present  a  discussion  of  the  general  rationale  for  inflation  and 
price  indexes,  whether  applied  to  the  economy  as  a  whole,  to  the  government,  or  to  the 
DoD.  In  the  section  The  Derivation  of  Inflation  Indexes  for  Use  by  the  DoD,  we  describe  how 
DoD  price  indexes  are  developed.  We  address  (a)  the  regulatory  and  statutory  provisions 
that  govern  the  issuance  of  inflation  guidance  by  the  Under  Secretary  of  Defense, 
Comptroller  (USD[C]),  and  (b)  how  these  provisions  are  applied  by  describing  the  key 
features  of  the  processes  used  in  the  Office  of  the  Secretary  of  Defense  (OSD)  and  in  the 
Services  to  produce  inflation  guidance. 

In  the  next  two  sections,  we  turn  to  how  the  DoD  uses  the  deflators  and  other 
considerations  in  budgeting  and  in  cost  analyses  related  to  procurement.  We  then  discuss 
the  current  practices  by  the  DoD  in  general  and  by  the  Services.  In  the  section  Analysis  of 
Alternative  Deflators  for  MDAPs,  we  compare  the  USD(C)’s  price  index  for  procurement  with 
alternatives,  principally  the  national  defense  indexes  published  by  the  Bureau  of  Economic 
Analysis  (BEA),  and  defense-related  relevant  producer  price  indexes  published  by  the 
Bureau  of  Labor  Statistics  (BLS).  The  purpose  of  these  comparisons  is  to  explore  the 
possibility  that  modifications  to  current  practices  might  better  meet  the  DoD’s  needs  for 
realistic  cost  estimation. 

We  next  assess  current  DoD  practices  for  accounting  for  inflation;  and  in  the  final 
section,  we  present  concluding  observations  and  recommendations. 

We  are  careful,  in  discussing  price  indexes,  to  differentiate  between  those  that  cover 
the  entire  economy  and  those  that  cover  specific  classes  of  goods  and  services.  The  former 
we  generally  refer  to  as  inflation  indexes  and  the  latter  as  price  indexes  or  escalation 
indexes. 

The  General  Rationale  for  Inflation  Indexes 

The  purpose  of  inflation  and  other  price  indexes  is  to  relate  changes  in  the  quantity 
of  resources  bought  or  sold  to  the  amount  of  money  spent  on  them  (Allen,  1935,  p.  58). 

Price  indexes  identify  and  isolate  the  effect  of  price  changes.  Removing  the  effect  of  price 
changes  leaves  information  on  quantity,  or  real,  changes.  Indexes  permit  us  to  answer 
questions  like  the  following1: 


1  Conceptually,  a  price  index  measures  the  ratio  of  expenditures  under  two  alternative  price  systems 
that  provide  quantities  of  goods  and  services  of  the  same  value. 
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•  What  has  been  the  change  in  the  real  size  of  the  economy  over  time? 

•  What  effect  have  changes  in  the  DoD  budget  had  on  the  resources  taken 
from  the  economy  and  the  resources  available  to  the  DoD? 

•  How  much  real  cost  growth  has  there  been  in  particular  DoD  procurement 
programs? 

Price  indexes  are  meant  to  capture  changes  in  the  price  of  a  particular  level  of 
capability.  They  should  not  capture  price  changes  that  are  due  to  changes  in  the  quality  of 
products.  As  an  example,  the  availability  of  much  better  computers  at  only  slightly  higher 
prices  means  society  has  gotten  richer  in  real  terms.  Allowing  price  indexes  to  rise  with  price 
increases  associated  with  quality  improvements  would  make  this  appear  not  to  be  the  case, 
so  price  indexes  should  not  reflect  the  price  of  quality  improvements.  In  other  words,  the 
portion  of  price  changes  that  reflect  quality  improvements  should  be  subtracted  from  price 
indexes.  (We  will  later  see  that  BEA  and  BLS  indexes  follow  this  procedure.) 

Price  indexes  can  be  developed  for  different  classes  of  goods  and  services:  the 
economy  as  a  whole;  all  DoD  spending;  DoD  procurement;  specific  types  of  DoD  goods 
such  as  aircraft,  ships,  and  computers;  and  the  input  prices  facing  firms  that  produce  things 
for  the  DoD.  Price  indexes  for  different  kinds  of  goods  and  services  can  vary  substantially 
overtime.  Figure  1  shows  how  indexes  for  commercial  goods  and  services  have  varied  with 
the  type  of  good  and  over  time  during  the  last  40  years.  Some  types  of  goods  and  services 
have  moved  along  with  the  overall  consumer  price  index  (CPI).  For  example,  the  price  of 
apparel  has  risen  far  more  slowly,  and  the  price  of  medical  care  has  climbed  at  nearly 
double  the  overall  rate  since  1970. 


iHasAII  items 
Food 
Apparel 
Housing 
Transportation 
■  Medical  care 
E  nergy 


Figure  1.  Consumer  Prices  for  Selected  Classes  of  Major  Expenditures 

(Administration  of  Barack  Obama,  n.d.,  Table  B-60) 


The  fact  that  one  index  has  not  fit  all  cases  of  commercial  goods  suggests  that 
budgeting  defense  goods  for  the  future  should  also  distinguish  between  types  of  goods.  In 
an  International  Monetary  Fund  paper,  Premchand  (1983)  put  it  succinctly:  “Every  budget  is 
formulated,  either  explicitly  or  implicitly,  on  a  price  basis.  As  prices  rise  and  become 
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relatively  unpredictable,  the  problems  of  budgeting  are  felt  more  keenly”  (p.  242).  Using 
different  price  indexes  for  different  goods  can  help  to  ameliorate  these  problems.  The  BEA, 
which  produces  the  U.S.  National  Income  and  Product  Accounts,  notes  that  the  use  of  a 
price  index  is  appropriate  if  its  definition  and  coverage  closely  match  the  category  of  product 
to  which  it  is  applied  (BEA,  2009). 

Different  organizations  take  different  approaches  in  accounting  for  inflation  in 
budgeting.  Organizations  such  as  the  Treasury  and  the  Office  of  Management  and  Budget 
(OMB)  that  are  involved  in  financing  aggregate  government  expenditure  focus  on  broad 
issues,  such  as  the  balance  between  the  public  and  private  sectors,  and  particularly  on  the 
value  to  the  private  sector  of  resources  taken  for  public  purposes.  These  offices  commonly 
analyze  these  issues  using  the  GDP  deflator,  an  index  based  on  the  price  of  the  market 
basket  of  all  goods  and  services  provided  to  final  users  by  the  entire  U.S.  economy.2  By 
comparison,  organizations  such  as  the  DoD  Comptroller’s  office  that  are  responsible  for  the 
budgets  of  particular  government  agencies  frequently  use  indexes  that  reflect  the  prices  of 
the  specific  resources  their  agencies  buy  to  support  their  activities  (Premchand,  1983,  pp. 
246-247).  A  possible  compromise  would  use  specific  indexes  to  develop  budgetary 
requirements  and  a  broad  index  to  reflect  the  constant-dollar  burden  implied  for  the 
economy  as  a  whole. 

The  Derivation  of  Inflation  Indexes  for  Use  by  the  DoD 

This  section  has  three  objectives: 

•  to  identify  the  regulatory  and  statutory  provisions  that  authorize  and  prescribe 
the  issuance  and  use  of  guidance  related  to  inflation  in  the  DoD; 

•  to  describe  the  flow  of  information  for  developing  the  economic  assumptions, 
including  those  for  inflation,  used  in  generating  the  President’s  Budget;  and 

•  to  describe  the  five  price  indexes  constructed  by  OMB  and  how  they  are  used 
to  develop  the  Comptroller’s  appropriation-specific  deflators. 

Regulatory  and  Statutory  Basis 

The  statutory  requirement  for  all  government  budgeting  is  contained  in  Title  31  of  the 
United  States  Code  (U.S.C.),  §  1104,  entitled  “Money  and  Finance.”  This  title  directs  the 
President  to  create  an  annual  budget,  delegating  administrative  authority  to  the  OMB.3  The 
OMB  requires  every  agency  to  prepare  an  annual  budget  for  its  spending  that  expresses  the 
administration’s  most  recent  policy  objectives  (31  U.S.C.  §  11 09). 4  OMB  forms  these  inputs 
into  a  total  annual  “policy”  budget  called  the  President’s  Budget. 

The  President’s  Budget  consists  of  spending  for  two  types  of  programs: 

•  discretionary  programs,  such  as  DoD  procurement  line  items,  which  are 
funded  at  a  level  decided  by  Congress  every  year;  and 


2  GDP  is  the  sum  of  consumption,  investment,  government  spending,  and  exports  minus  imports. 

3  31  U.S.C.  §  1 1 04  resulted  from  the  Budget  and  Accounting  Act  of  1 921 .  Administrative  responsibility 
initially  existed  within  the  Bureau  of  the  Budget,  with  the  OMB  tasked  through  Executive  Order  in 
1970. 

4  The  OMB  also  prepares  a  “baseline,”  or  “current  services”  budget,  that  assumes  that  current-year 
programs  will  extend  into  the  budget  year  and  out-years,  and  updates  their  costs  using  the  most 
recent  economic  assumptions. 
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•  mandatory  programs,  such  as  Social  Security  and  Medicare,  which  are 
passed  as  permanent  law  by  congressional  authorization,  written  into  the 
U.S.C,  and  funded  by  annual  appropriation  as  directed  by  the  permanent  law. 

This  paper  concerns  inflation  for  only  the  discretionary  programs.  The  following 
paragraphs  describe  the  general  guidance  contained  in  OMB  Circulars  A-1 1  and  A-94  and 
the  specific  guidance  to  DoD  Components  in  the  Financial  Management  Regulation  (FMR), 
issued  by  the  OUSD(C),  for  meeting  the  OMB  guidance. 

OMB  Circular  No.  A-1 1  (2009)  sets  policy  for  how  agencies  are  to  treat  inflation  in 
their  budget  requests  submitted  for  executive  review.  The  relevant  excerpt  from  Section  31 
(Paragraph  31.1(c))  of  the  circular  provided  below  states  that  agencies  must  ensure  that 
their  inputs  to  the  discretionary  part  of  their  budgets  must  be  consistent  with  the  OMB’s 
economic  assumptions,  including  those  relating  to  inflation. 

(c)  What  economic  assumptions  should  I  use  when  I  develop  estimates? 

All  budget  materials,  including  those  for  the  outyear  policy  and  baseline 
estimates,  must  be  consistent  with  the  economic  assumptions  provided  by 
OMB.  The  specific  guidance  below  applies  to  outyear  policy  estimates. 

OMB  policy  permits  consideration  of  price  changes  for  goods  and 
services  as  a  factor  in  developing  estimates.  However,  this  does  not  mean 
that  you  should  automatically  include  an  allowance  for  the  full  rate  of 
anticipated  inflation  in  your  request.  ... 

For  discretionary  programs,  you  may  include  an  allowance  for  the  full 
rate  of  anticipated  inflation,  an  allowance  for  less  than  the  full  rate,  or  even  no 
allowance  for  inflation.  In  many  cases,  you  must  make  trade-offs  between 
budgeting  increases  for  inflation  versus  other  increases  for  programmatic 
purposes.5 

OMB  Circular  No.  A-94  (1992)  provides  agencies  with  guidance  for  cost-benefit 
analyses.  It  recommends  using  the  gross  domestic  product  (GDP)  deflator  for  the  overall 
inflation  rate — the  general  increase  in  prices  of  goods  and  services — but  permits  using 
sector-specific  indexes  that  differ  from  the  general  inflation  rate  “where  there  is  a  reasonable 
basis  for  estimating  such  changes.”  Projects  with  a  budget  horizon  longer  than  six  years  (the 
Future  Years  Defense  Program  (FYDP)  years  in  the  case  of  the  DoD)  are  advised  to  use  the 
final  year’s  rate  in  perpetuity. 

The  Financial  Management  Regulation  (FMR)  7000  14-R  (DoD,  n.d.)  provides 
inconsistent  guidance  concerning  price  indexes  in  two  paragraphs  of  Volume  2A,  Chapter  1, 
Section  010303.  Paragraph  B.1  states  that  DoD  budget  estimates  should  “reflect  the  most 
likely  or  expected  full  costs.”  Paragraph  B.2,  however,  mandates  that  “price  level  changes 
will  be  based  on  data  provided  by  OUSD  (Comptroller),”  and  that  the  Comptroller’s 
appropriation-specific  price  indexes  should  be  used  to  “determine  the  amount  of  price 
escalation  for  a  procurement  line  item,  major  RDT&E  system,  or  construction  item  over  a 
given  time  period.”  This  guidance  is  being  revised  to  make  it  clear  that  the  most  likely  or 
expected  full  costs  in  then-year  dollars  should  be  used  in  budget  preparation — even  if  this 
implies  price  increases  different  from  those  implied  by  Comptroller’s  indexes — and  that 
Comptroller  indexes  must  be  used  to  convert  then-year  dollar  values  to  constant-dollar 
values. 


5  This  section  is  titled  “Compliance  With  Administration  Policies  and  Other  General  Requirements” 
and  is  the  only  inflation  guidance  that  appears  in  the  1 ,000-page  document. 
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Paragraph  B.2  seems  to  direct  the  use  of  the  Comptroller’s  indexes  as  the  only 
acceptable  value  for  calculating  price  escalation  for  specific  programs,  while  the  “most  likely 
or  expected  full  costs”  of  paragraph  B.1  are  presumably  those  for  the  specific  items  being 
purchased  (DoD,  n.d.).  This  appears  inconsistent  because  the  Comptroller’s  indexes  are  not 
at  all  specific  to  the  particular  goods  being  purchased. 

Development  of  Economic  Assumptions 

Each  fall,  senior  officials  and  staff  from  the  OMB,  the  Council  of  Economic  Advisors, 
and  the  Department  of  the  Treasury  (collectively  known  as  the  “Troika”)  draw  on 
administration  policies  and  use  various  forecasting  models  to  produce  a  1 0-year  forecast  of 
key  economic  indicators,  including  inflation.  These  economic  assumptions  update  previous 
assumptions  to  reflect  recent  data.  They  are  used  in  forming  budget  outlay  and  revenue 
estimates  and  developing  the  annual  President’s  Budget. 

The  OMB  provides  the  economic  assumptions  regarding  inflation6  to  the  federal 
agencies  each  November  as  guidance.  That  guidance,  and  how  the  DoD  Comptroller  uses  it 
to  develop  more  detailed  guidance  for  DoD  Components,  is  discussed  next. 

Derivation  of  Appropriation-Specific  Price  Indexes 

OMB  guidance  sent  to  the  OUSD(C)  covers  the  two  prior  years,  the  budget  year,  and 
four  out-years  for  five  categories  of  funding: 

•  Military  pay,  using  the  projected  Employment  Cost  Index  (ECI)  for  wages  and 
salaries  published  by  the  BLS,  of  the  Department  of  Labor,  adjusted  for 
administration  policy  recommendations  as  prescribed  in  Title  37  U.S.C. 
Section  1009 

•  Civilian  pay,  using  the  projected  ECI  less  0.5  percentage  points,  adjusted  for 
administration  policy  recommendations,  as  prescribed  in  Title  5  U.S.C. 
Section  5303 

•  Fuel,  using  the  projected  Energy  Information  Administration  Refiner 
Acquisition  Cost;  this  is  the  oil  refiners’  average  price  for  crude  oil 

•  Medical,  using  the  projected  BLS  Consumer  Price  Index  for  All  Urban 
Consumers  (CPI-U)  Medical  price  index 

•  Other  purchases — all  purchases  other  than  the  four  categories  just  listed — 
using  the  projected  values  of  BEA’s  GDP  price  index  as  determined  by  the 
Troika  and  provided  to  the  Comptroller  by  the  OMB 

The  OUSD(C)  uses  weighted  averages  of  these  five  OMB  indexes  to  construct  the 
annual  price  indexes  (often  called  deflators)  for  the  DoD  appropriation-level  accounts  shown 
in  Table  1 .  The  weights  are  based  on  how  the  spending  for  each  account  is  distributed 
across  the  resources  represented  by  the  OMB  indexes  (military  pay,  civilian  pay,  etc.). 


6  The  Administration’s  economic  assumptions  include  projections  of  consumer  inflation  measured  by 
the  urban  Consumer  Price  Index,  GDP  (Current,  Real,  and  the  Price  Index  between  them), 
Unemployment  rate,  91-day  Treasury  Bill  interest  rate,  and  10-year  Treasury  Bill  interest  rate.  They 
are  available  in  the  OMB’s  “Supplemental  Materials”  (see  OMB,  n.d.). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-412- 


Table  1.  Composition  of  Appropriation  Level  Inflation  Deflators 

(Office  of  the  Undersecretary  of  Defense,  Comptroller  [OUSD(C)],  2010) 


Appropriation 
(FY  2010  Outlays) 

MilPay 

CivPay 

Fuel  Medical 

Other 

Purchases 

Military  Personnel  (S155.0B) 

61% 

8% 

31% 

O&M  (S279.7B) 

30% 

5%  12% 

53% 

Procurement  (S147.2B) 

100% 

RDT&E  (S79.3B) 

11% 

<1% 

89% 

Military  Construction  (S23.8B) 

5% 

95% 

Family  Housing  ($3.3B) 

4% 

1% 

95% 

The  OMB  directs  that,  in  deflating  program  spending  for  years  beyond  those  for 
which  indexes  have  been  made  available,  program  managers  should  extend  the  final  year’s 
inflation  rate  into  the  later  years  (OMB,  1992,  Section  7.b.). 

The  table  illustrates  the  process  for  the  FY  201 0  budget.  For  example,  30%  of  total 
DoD  spending  on  operations  and  maintenance  (O&M)  was  for  civilian  pay.  The  O&M  index 
was  therefore  calculated  as  follows: 

O&M  index  =  (CivPay  index)  x  0.30  +  (Fuel  index)  x  0.05 
+  (Medical  index)  x  0.12  +  (Other  Purchases  index)  x  0.53 

It  is  significant  that  while  the  first  four  OMB  indexes  characterize  specific  types  of 
resources  (civilian  pay,  etc.),  the  last  one,  “other  purchases,”  does  not.  In  fact,  the  OMB 
index  for  all  other  purchases  is  the  GDP  deflator,  the  single  price  index  for  all  spending  on 
U.S.  goods  and  services.  The  GDP  deflator  is  the  main  determinant  of  the  amount  of 
inflation  allowed  in  the  DoD  budget.  It  is  the  sole  determinant  for  procurement  spending  and 
is  applied  to  fully  64%  of  total  spending.  (Weighting  the  “other  purchases”  percentages  in 
the  last  column  of  Table  1  by  the  proportion  of  total  outlays  implied  in  the  first  column  yields 
a  weighted  average  of  64%. ) 

The  OUSD(C)  deflators  are  issued  to  the  DoD  Components  by  guidance  memo.  The 
assistant  secretary  (financial  management  and  comptroller)  of  each  military  department 
issues  implementing  guidance  to  its  commands  and  components  that  is  tailored  to  its 
department’s  administrative  procedures.  The  components  use  the  deflators  and  instructions 
contained  in  the  DoD  FMR  to  re-price  the  President’s  Budget  through  a  resource 
management  decision  for  submission  to  the  OMB,  and  also  to  prepare  detailed  budget 
justification  material  for  submission  to  the  Congress. 

Current  Practice  for  Incorporating  Inflation  Into  Program  Budgets  and  Cost  Estimates 
for  Major  Defense  Acquisition  Programs 

The  DoD  buys  millions  of  different  products:  food  for  Service  mess  halls,  spare  parts, 
construction  material,  medical  supplies,  medical  equipment,  construction  equipment,  and 
many  others.  In  these  instances,  the  DoD  buys  at  prices  generally  available  in  the  market  to 
large  buyers.  Price  indexes  for  these  kinds  of  commodities  are  properly  based  on  their 
output  prices.  Such  indexes  might  often  approximate  a  broad-based  index  like  the  GDP 
deflator. 
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In  this  paper,  we  do  not  focus  on  these  kinds  of  purchases.  We  are  interested 
specifically  in  MDAPs  because  they  are  the  focus  of  WSARA.  Contracting  procedures 
require  that  the  prices  of  major  defense  systems  be  based  on  the  costs  of  the  inputs  to  the 
systems:  labor  and  materials.  This  is  even  true  of  fixed-price  types  of  contracts.  Firm-fixed- 
price  contracts  are  based  on  the  expected  cost  of  inputs,  while  fixed-price  with  economic 
price  adjustment  contracts  incorporate  fluctuations  in  labor  or  material  costs  during  the 
period  of  contract  performance.  It  appears  that  the  use  of  price  indexes  based  on  the 
relevant  input  prices  is  best  for  MDAPs. 

In  this  section,  we  provide  an  overview  of  the  treatment  of  inflation  by  MDAPs  and 
then  turn  to  the  practices  of  the  individual  Services. 

General  Considerations  in  Use  of  Inflation  Indexes  by  Program  Managers 

Program  budgeters  have  to  think  about  inflation  for  two  reasons: 

•  In  budgeting,  they  must  estimate  the  future  costs  of  their  procurement 
programs  in  then-year  dollars  that  are  based  on  expected  increases  in  prices. 

•  They  must  calculate  real  cost  increases  of  systems  being  acquired  in 
constant  (inflation-corrected)  dollars,  also  termed  real  cost  growth.  Such 
calculations  are  used  to  identify  systems  that  are  suffering  from  high  levels  of 
real  cost  growth,  a  focus  of  WSARA. 

In  addition,  all  parts  of  the  DoD  must  use  price  indexes  to  translate  budget 
submissions  developed  in  then-year  dollars  to  constant-dollar  terms. 

Regarding  budgeting,  for  a  program  to  be  fully  funded,  money  must  be  appropriated 
up  front  to  cover  all  projected  future  then-year  costs  of  the  portion  of  the  program  authorized 
in  a  given  year,  such  as  a  specified  annual  production  lot.  If  planners  underestimate  the 
extent  to  which  the  cost  of  the  authorized  program  will  rise  over  time,  due  to  either 
unanticipated  general  inflation  or  increases  in  the  prices  of  inputs  specific  to  the  program, 
appropriations  will  fall  short  and  an  overrun  will  occur — an  undesirable  outcome.  We  noted 
earlier  that  guidance  regarding  the  treatment  of  inflation  in  budgeting  appears  inconsistent, 
calling  for  the  use  of  OUSD(C)  deflators  and  also  mandating  use  of  “most  likely  or  expected 
full  costs.”  As  we  shall  see,  some  DoD  organizations  rely  on  the  Comptroller’s  projections  of 
inflation  for  developing  then-year  budget  estimates,  while  others  do  not. 

Real  cost  growth  is  measured  by  the  percentage  increase  in  unit  cost  relative  to  a 
past  baseline  evaluated  in  baseline-year  constant  dollars.  The  baseline  cost  can  be  either 
the  original  program  cost  or  a  later  estimate,  depending  on  the  program’s  history.  For 
procurement  programs,  the  Nunn-McCurdy  Amendment  to  the  1982  National  Defense 
Authorization  Act  requires  the  DoD  to  identify  for  special  attention  those  programs  whose 
average  unit  cost  growth  has  breached  stated  thresholds. 

Selected  Acquisition  Reports  (SARs)  are  used  as  the  source  of  information 
concerning  cost.  The  GDP  deflator  is  always  used  to  convert  current-dollar  costs  to  constant 
base-year  dollars  both  for  establishing  the  real  cost  baseline  and  for  calculating  real  cost 
growth. 

We  now  turn  to  the  specifics  of  how  various  DoD  organizations  incorporate  inflation 
into  their  program  budget  estimates. 

Practices  of  Individual  Organizations 

In  this  section,  we  briefly  describe  the  procedures  various  DoD  organizations  use  in 
incorporating  inflation  into  program  procurement  budgets.  Information  in  this  section  is 
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based  on  discussions  with  staff  in  the  organizations  cited.  Because  not  all  relevant 
organizations  have  been  contacted,  this  is  not  a  complete  survey. 

Army 

The  Army  follows  OSD  budget  guidance  without  exception  in  adjusting  program 
costs  and  budgets  for  inflation.7  The  indexes  the  Army  uses  are  stored  together  with  the 
standard  Navy  and  Marine  Corps  indexes  on  the  Navy  Center  for  Cost  Analysis’s  (NCCA’s; 
n.d.)  website  tool  for  calculating  inflation  factors. 

Navy  and  Marine  Corps 

NAVSEA  Projections  of  Shipbuilding  Cost.  The  Naval  Sea  Systems  Command 
(NAVSEA)  follows  a  systematic  methodology  to  develop  its  own  estimates  of  inflation  for 
budgeting  its  ship  programs.  NAVSEA  developed  this  methodology  in  response  to  a  2004 
direction  from  the  Under  Secretary  of  the  Navy  for  Acquisition. 

NAVSEA  has  developed  a  complex  and  detailed  model  for  making  these  estimates 
based  on  current  and  historical  data  on  labor  and  material  inputs.  Labor  prices  reflect 
shipyard-specific  labor  and  overhead  rates  based  on  shipbuilder  forward  pricing  rate 
agreements  (FPRAs).8  Material  prices  include  class-specific  material  inflation  and  vendor 
base  adjustments  unique  to  each  ship  type’s  market  sector  (nuclear,  non-nuclear, 
commercial,  etc.).  Estimates  of  future  prices  are  based  on  forecasts  by  Global  Insight,  a 
private  firm  that  has  been  involved  in  economic  and  financial  analysis  and  forecasting  for 
many  years.  Historical  indexes  for  labor  cost  increases  are  based  on  actual  shipyard  data, 
aggregated  to  the  national  level  based  on  the  workload  at  each  shipyard.  Historical  material 
indexes  are  based  on  BLS  producer  price  indexes. 

NAVSEA’s  projections  of  shipbuilding  cost  increases  are  higher  than  the 
procurement  cost  forecasts  issued  by  OUSD(C).  NAVSEA  estimated  annual  shipbuilding 
inflation  at  3.3%  during  2010-2015,  while  the  OUSD(C)  procurement  index  (the  GDP 
deflator)  increased  at  an  average  annual  rate  of  only  1.5%. 

NAVAIR  Pricing  Models.  The  Naval  Air  Systems  Command  (NAVAIR)  develops  its 
own  projections  for  pricing  naval  aircraft  (fixed-  and  rotary-wing).  In  a  similar  fashion  to  the 
NAVSEA  model,  NAVAIR  develops  estimates  for  labor  and  material  cost  increases  and  uses 
these  to  develop  estimates  for  airframe,  engine,  and  electronics,  which  are  then  combined 
into  an  overall  estimate  for  fixed-wing  aircraft  flyway  cost. 

The  variance  in  these  year-to-year  projections  is  surprising.  Note,  for  example,  that 
aircraft  inflation  is  forecast  to  be  halved  from  2015  to  2016. 

NAVAIR  also  makes  detailed  projections  for  helicopters  and  missiles.  Future  labor 
rates  are  based  on  projections  for  the  labor  contracts  of  the  major  aircraft  and  missile 
manufacturers,  and  materials  prices  are  derived  from  estimates  by  Global  Insight. 

Marine  Corps.  Marine  Corps  policy  is  to  use  the  prescribed  OUSD(C)  inflation 
factors  for  program  budget  and  cost  estimates.  No  exceptions  have  been  identified. 

Air  Force 

Air  Force  policy  for  inflation  adjustments  is  decentralized,  unlike  that  of  the  Army  and 
Navy.  Program  offices  may  develop  their  own  inflation  projections  using  industry-specific 


7  Discussion  with  personnel  in  the  Army  Cost  Analysis  Agency. 

8  An  FPRA  is  a  written  agreement  negotiated  between  a  contractor  and  the  government  to  use  certain 
rates  during  a  specified  period  for  pricing  future  contracts  or  modifications. 
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prices.  These  estimates,  however,  are  subject  to  review  by  program  executive  officers, 
Service  acquisition  executives,  the  Air  Force  Cost  Analysis  Agency  (AFCAA),  and  the 
pertinent  OSD  offices.  The  description  below  is  based  on  personal  communication  from  the 
staff  of  AFCAA  and  other  organizations. 

Air  Force  Aircraft.  Most  Air  Force  aircraft  program  offices  estimate  future  program 
costs  using  specific  inflation  rates  obtained  by  combining  labor  and  material  price  rates, 
commercial  forecasting  model  estimates,  and  contract  information  on  FPRAs.  The  methods 
they  use  appear  similar  to  those  adopted  by  NAVAIR. 

Space  Systems.  Most  programs  use  specific  rates  developed  from  historical  data  on 
inflation  in  space  systems  and  comparisons  with  general  inflation. 

Information  Technology.  Most  programs  appear  to  use  OUSD(C)-promulgated 

rates. 


National  Reconnaissance  Office 

The  National  Reconnaissance  Office  (NRO)  purchases  optical-  and  radar-imaging 
satellites  for  reconnaissance  and  surveillance  missions.  The  NRO  in  2004  compared  its 
contractors’  labor  and  material  prices  with  the  standard  inflation  guidance  for  1995-2001. 
Labor  prices  increased  by  4.2%  per  year  on  average,  but  material  prices  showed  no  upward 
trend.  Combining  the  labor  and  material  prices  with  the  appropriate  weights  yielded  an 
average  annual  inflation  rate  of  3%.  The  OUSD(C)  procurement  deflator  increased  by  1.4% 
annually  during  the  same  period  (Odom,  2004).  The  NRO  bases  its  budget  and  cost 
estimates  in  large  part  on  Global  Insight  direct  labor  and  material  price  indexes.9 

Summary 

We  have  seen  that  some  DoD  organizations  develop  specialized  inflation  indexes  for 
their  programs  and  use  them  to  ensure  that  their  budget  submissions  “reflect  most  likely  or 
expected  full  costs.”  These  indexes  are  used  both  for  development  of  cost  estimates  for 
programs  in  then-year  dollars  and  for  budgeting.  These  rates  can  be  substantially  higher 
than  those  provided  by  the  OMB. 

Real  program  cost  and  cost  growth  for  MDAPs  are  then  calculated  using  the  GDP 
deflator  to  convert  current  dollars  to  constant  dollars. 

We  now  turn  to  a  comparison  of  the  OUSD(C)  price  index — the  GDP  deflator — with 
other  alternatives  developed  by  BEA  and  BLS.  Our  interest  here  is  in  seeing  whether  using 
price  indexes  tailored  to  different  defense  goods  such  as  aircraft  and  ships  might  offer  DoD 
better  tools  for  accounting  for  inflation. 

Analysis  of  Alternative  Deflators  for  MDAPs 

Introduction 

Note  by  way  of  background  that  all  DoD  procurement  outlays,  including  MDAPs, 
account  for  less  than  1  %  of  the  GDP.  There  is  no  particular  reason  to  believe  that  DoD 
procurement  prices  move  in  tandem  with  the  other  99%  of  the  economy.  Moreover,  using  a 


9  We  have  not  comprehensively  surveyed  the  defense  agencies  or  other  organizations  to  establish 
their  policies  with  respect  to  projecting  inflation.  Most  such  organizations  do  not  have  substantial 
procurement  budgets.  Those  that  do  have  substantial  procurement  budgets  include  the  Special 
Operations  Command,  the  Defense  Communications  Agency,  and  the  National  Security  Agency,  but 
we  do  not  have  information  for  them. 
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single  price  index  for  all  MDAPs  ignores  the  differences  among  the  various  military  goods 
that  are  procured  and  the  markets  from  which  they  are  bought. 

We  proceed  by  first  comparing  the  distribution  of  DoD  purchases  with  those  in  the 
economy  as  a  whole  and  then  comparing  DoD  inflation  for  various  procurement  categories 
with  other  inflation  indexes  of  possible  interest  and  with  the  GDP  deflator.  After  that,  we 
consider  the  issue  of  accurately  forecasting  inflation. 

The  Distribution  of  Spending  Across  Economic  Sectors 

Figures  from  Inforum  show  that  the  top  10  sectors  that  the  DoD  buys  from  are,  with 
the  exception  of  wholesale  trade,  all  different  from  the  top  10  sectors  for  the  economy  as  a 
whole.10  The  10  sectors  account  for  roughly  half  of  all  purchases  in  both  categories, 
excluding  direct  purchases  of  labor. 

Because  the  DoD  and  the  overall  economy  purchase  very  different  mixes  of  items, 
using  the  GDP  deflator  to  represent  price  changes  for  defense  purchases  is  questionable. 
Alternative  price  indexes  might  provide  a  better  representation. 

Retrospective  Comparison  of  GDP  With  Alternative  Price  Indexes 

Bureau  for  Economic  Analysis  National  Defense  Deflators 

In  addition  to  the  GDP  price  deflator,  the  BEA  publishes  deflators  for  procurement  of 
five  major  types  of  military  systems:  aircraft,  missiles,  ships,  vehicles,  and  electronics. 

Figure  2  and  Table  2  compare  these  defense  deflators  to  the  GDP  deflator  during  the  1985- 
2009  time  period.* 11 

The  defense  deflators  are  “quality  adjusted”  to  measure  price  changes,  holding  the 
physical  specifications  of  the  systems,  or  their  “quality,”  constant.  Examples  of  quality 
adjustment  for  aircraft  are  features,  such  as  engine  improvements.  The  BEA  measures  the 
value  of  quality  changes  by  their  cost  of  production  and  excludes  them  from  the  price  index 
by  subtracting  the  average  quality  production  cost  from  the  average  total  production  cost 
(Ziemer  &  Kelly,  1993;  Foss,  Manser,  &  Young,  1993).  The  BEA  deflator  is  thus  influenced 
by  changes  in  average  cost  due  to  factors  other  than  improved  specifications,  such  as 
changes  in  input  prices.  According  to  the  BEA,  it  may  be  difficult  to  estimate  the  quality 
change  when  an  entirely  new  kind  of  aircraft,  such  as  UAVs,  is  introduced,  leading  them  to 
consider  the  entire  price  as  quality  change. 


10  The  figures  are  from  360  sector  databases  developed  by  Inforum  (The  Interindustry  Forecasting 
Project  at  the  University  of  Maryland).  The  DoD  figures  are  from  the  “Federal  Defense”  table  and  the 
economy-wide  figures  are  from  the  “National”  table.  (“National”  combines  spending  for  federal 
defense,  federal  non-defense,  non-federal  government,  and  the  private  sector;  Inforum,  n.d.). 

11  These  BEA  deflators  are  expenditure-weighted  averages  of  separate  deflators  for  durables  (largely 
spares,  modifications,  overhauls,  and  support  equipment)  and  gross  investment  (new  equipment). 
The  data  are  from  BEA  National  Income  and  Product  Accounts  Table  3.1 1.4,  Price  Indexes  for 
National  Defense  Consumption  and  Gross  Investment  (BEA,  2013). 
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Figure  2.  GDP  vs.  BEA  National  Defense  Deflators 


Table  2.  Comparison  of  BEA  National  Defense  Deflators 


Deflator 

Average  Annual 
Growth  Rate 
1985-2009 

Total  Growth 
1985-2009 

Defense  Ships 

2.7% 

90% 

GDP 

2.4% 

78% 

Defense  Vehicles 

1.9% 

56% 

Defense  Aircraft 

0.1% 

1% 

Defense  Missiles 

-0.3% 

-8% 

Defense  Electronics 

-1.5% 

-31% 

The  BEA  deflators  in  Figure  2  show  wide  variation:  (a)  substantial  deflation  over  the 
period  for  electronics  (which  includes  software),  (b)  virtually  no  change  in  the  indexes  for 
aircraft  and  missiles,  and  (c)  substantial  inflation  for  ships  and  vehicles.  The  large  decline  for 
electronics  is  due  to  the  fact  that  computer  speed,  memory,  and  storage  capacity  have  been 
rising  faster  than  price  for  many  years.  The  table  and  figure  show  that  all  of  the  BEA  national 
defense  deflators  except  for  ships  have  had  measurably  to  substantially  less  growth  than 
the  GDP  deflator  over  the  period.  The  wide  variations,  however,  may  be  due  to  how  the  BEA 
identifies  and  measures  quality  adjustments. 

Bureau  of  Labor  Statistics  Producer  Price  Indexes 

Figure  3  and  Table  3  compare  the  GDP  price  deflator  with  the  producer  price 
indexes  (PPIs)  that  the  BLS  publishes  for  military  and  analogous  commercial  systems.  Like 
the  BEA  deflators,  BLS  price  indexes  are  quality  adjusted.  The  algorithms  are  described 
differently  but  are  mathematically  equivalent,  and  they  employ  the  same  general  criteria 
(holding  specification  constant).  However,  there  is  no  communication  between  the  two 
organizations  on  how  DoD  procurement  data  are  handled. 

The  bottom  four  PPIs  in  Figure  3  (solid  lines  other  than  for  the  GDP  deflator)  are 
relevant  to  defense,  and  the  top  three  (dashed  lines)  are  for  analogous  civilian  goods 
included  for  comparison.  The  PPIs  show  substantially  smaller  growth  rates  for  military 
aircraft  engines  and  ships  than  for  the  analogous  civilian  goods.  The  disparity  between  the 
GDP  and  military  growth  rates  is  less  for  the  PPIs  than  for  the  BEA  national  defense 
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deflators  shown  earlier.  Aircraft  engines  have  grown  less,  ships  have  grown  about  the  same, 
and  aerospace  goods  have  grown  more.  (We  are  regarding  the  aerospace  PPI  as  reflecting 
defense  goods  because  BLS  includes  military  communication  and  reconnaissance  satellites 
as  well  as  civilian-funded  NASA  space  shuttles.)  A  now-discontinued  PPI  deflator  for 
electronic  computers  during  the  1991-2003  time  period,  normalized  to  1991  =  100,  indicates 
that  computers  experienced  a  huge  average  annual  (quality  adjusted)  price  decrease  of 
14.8%  during  this  period  (Table  3). 


Figure  3.  Producer  Price  Index  Defense  and  Analogous  Civilian  Deflators 

(BLS,  n.d.) 


Table  3.  Comparison  of  Bureau  of  Labor  Statistics  Defense-Related  Deflators 


Deflator 

Average  Annual 
Growth  Rate 

Total  Growth 

PPI  Non-military  ship  construction 

4.4% 

182% 

PPI  Civilian  aircraft 

3.6% 

135% 

PPI  Civilian  aircraft  engines 

3.4% 

121% 

PPI  Aerospace  product  and  parts 

2.9% 

101% 

PPI  Military  ship  construction 

2.5% 

82% 

GDP 

2.4% 

78% 

PPI  Military  aircraft  engines 

1 .4% 

40% 

PPI  Electronic  computers  (1991-2003) 

-14.8% 

-85% 

As  with  the  BEA  deflators,  some  of  the  differences  in  growth  rates  might  be  due  to 
the  criteria  and  numerical  methods  for  making  quality  adjustments. 

Bureau  of  Economic  Analysis  and  Bureau  of  Labor  Statistics  Price  Indexes 
That  Are  Most  Relevant  for  Defense 

Figure  4  brings  together  the  BEA  and  BLS  PPI  series  that  are  most  relevant  to 
defense  final  products.  There  are  major  differences.  The  BEA  indexes  for  defense  aircraft, 
missiles,  and  electronics  have  grown  much  less  than  the  GDP  index.  The  aircraft  index  is 
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extremely  far  below  the  PPI  index  for  civilian  aircraft.  The  deflators  for  aerospace  and 
military  and  ships  are  quite  close  to  the  GDP  index.12 


Figure  4.  Defense  and  Producer  Price  Index  Deflators  Related  to  Defense 


Conclusions  From  Retrospective  Comparison  of  Alternative  Deflators 

The  BEA  national  defense  deflators  seem  most  relevant  to  MDAPs  because  of  the 
deflators’  focus  on  defense-related  products,  but  they  are  not  entirely  credible.  The  indexes 
for  aircraft  and  missiles  show  much  lower  rates  of  increase  than  the  GDP  deflator  and  even 
much  lower  rates  of  increase  than  is  measured  for  the  commercial  aircraft  sector.  As 
mentioned  earlier,  this  might  depend  in  part  on  how  costs  associated  with  improvements  in 
capability  are  measured  for  purposes  of  making  quality  adjustments.  Other  indexes — for 
example,  the  national  defense  deflators  for  ships  and  vehicles  and  the  PPI  for  military 
ships — have  moved  similarly  to  the  GDP  deflator. 

The  policy  implication  of  these  comparisons  is  that  the  difference  in  growth  rates 
among  the  defense  and  defense-related  indexes  suggests  that  the  DoD  might  obtain  better 
measures  of  the  real  value  of  the  overall  MDAP  budget  by  using  sector-specific  alternative 
price  indexes  instead  of  the  GDP  deflator.  However,  given  the  wide  variability  we  have 
observed,  our  analysis  fails  to  provide  a  clear  picture.  A  better  understanding  of  how  the 
quality  adjustments  are  made  is  needed. 

Perhaps  most  important,  neither  the  BEA  nor  the  BLS  provides  price  indexes  that  are 
derived  from  the  prices  of  inputs  used  in  the  production  of  various  types  of  MDAPs.  The 
development  and  use  of  such  indexes  by  organizations  like  NAVAIR  reflects  the  indexes’ 
superiority. 


12  The  BLS  does  not  publish  indexes  for  military  aircraft  because  there  are  not  enough  domestic 
producers  to  meet  the  BLS's  standards  for  survey  respondent  confidentiality  and  statistical  accuracy 
of  the  index. 
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Prospective  Analysis:  Success  in  Forecasting  Inflation 

Inflation  predictions  are  useful  in  budget  preparation  only  to  the  extent  that  they  are 
accurate.  The  OMB  forecasts  the  growth  rates  of  the  GDP  deflator  five  years  into  the  future, 
and  Figure  5  shows  the  accuracy  of  these  forecasts  during  the  past  19  years.  The  initial 
forecast  for  1991  in  1986,  for  example,  was  2.3%,  1.5%  lower  than  the  most  recent  estimate 
of  3.8%  in  2010. 

Overall,  the  five-year  forecasts  seem  fairly  accurate.  The  number  of  overestimates 
and  underestimates  was  about  the  same  (10  vs.  9),  and  the  absolute  value  of  the  yearly 
errors  averaged  only  0.8%.  The  overestimates  were  a  bit  larger  than  the  underestimates, 
with  maxima  of  1.7%  and  1.5%,  respectively. 


Figure  5.  Accuracy  of  Predictions  of  the  GDP  Inflation  Rate  Five  Years  in  the  Future 

(OUSD[C],  2010) 

The  estimates  usually  became  more  accurate  as  the  year  of  execution  approached, 
but  they  varied  a  good  deal  from  year  to  year. 

Because  organizations  like  NAVAIR  use  inflation  estimates  developed  by  Global 
Insight,  it  would  be  useful  to  examine  how  accurate  those  estimates  have  been. 
Unfortunately,  we  do  not  have  enough  information  at  present  to  conduct  such  an  analysis. 

Assessment  of  Current  Practices  for  Accounting  for  Inflation  in  the  DoD 

We  previously  mentioned  that  price  indexes  are  used  for  two  separate  purposes  in 
the  DoD: 

•  budgeting  for  future  spending,  and 

•  measuring  real  cost  growth  in  acquisition  programs  and  identifying  those 
programs  whose  real  cost  has  grown  enough  to  justify  special  management 
attention. 

A  key  goal  of  budget  development  for  particular  programs  is  to  allocate  sufficient  but 
not  excessive  funds  for  specific  purposes.  Budgeting  for  personnel,  fuel,  and  health-related 
expenses  draws  on  specific  price  indexes  tailored  for  them  and  should  meet  the  goal. 

In  the  case  of  MDAPs,  as  long  as  programs  follow  the  guidance  to  “reflect  most  likely 
or  expected  full  costs,”  the  goal  should  be  met.  However,  if  Comptroller  rates  are  used  to 
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estimate  future  price  increases,  in  cases  where  those  increases  are  expected  to  be  greater 
or  smaller  than  the  Comptroller  rates,  programs  will  be  underfunded  or  overfunded. 

Program  offices  may  have  a  tendency  to  over-estimate  future  price  increases  in 
order  to  build  contingency  reserves.  The  rationale  for  using  specific  price  indexes  should  be 
clearly  presented  in  budget  submissions  and  should  be  subject  to  systematic  review  and 
approval  at  both  the  Service  and  OSD  levels. 

Our  review  of  current  practices  in  the  section  Current  Practice  for  Incorporating 
Inflation  Into  Program  Budgets  and  Cost  Estimates  for  Major  Defense  Acquisition  Programs 
indicates  that  program-  or  sector-specific  price  indexes  based  on  input  prices  are  used  in 
shipbuilding,  aviation,  and  space — areas  in  which  Comptroller  rates  are  often  deemed  to 
rise  too  slowly.  The  section  Analysis  of  Alternative  Deflators  for  MDAPs  indicates  that  price 
increases  for  ground  vehicles  may  have  not  differed  greatly  from  the  GDP  deflator.  In  other 
words,  current  practices  for  procurement  budgeting  may  reflect  most  likely  or  expected  full 
costs  fairly  well  overall. 

Concerning  the  use  of  inflation  escalation  indexes  for  calculating  real  program  cost 
growth,  we  discuss  two  possibilities: 

•  Adjusting  for  changes  in  the  prices  of  inputs  used  for  the  particular  program. 
This  would  absolve  programs  of  responsibility  for  a  category  of  cost  increases 
that  are  largely  beyond  their  control. 

•  Adjusting  for  price  changes  in  the  economy  as  a  whole.  This  implies 
calculating  real  cost  growth  using  the  GDP  deflator. 

Use  of  program-specific  indexes  would  be  most  consistent  with  the  goal  of  identifying 
programs  whose  costs  have  risen  for  reasons  other  than  higher  input  prices.  However, 
program-specific  input  price  indexes  are  not  always  available,  and  there  is  some  virtue  in 
the  simplicity  of  using  a  single  index  to  calculate  real  cost  growth. 

Using  the  GDP  deflator  to  calculate  real  cost  growth  relative  to  the  baseline  can  be 
justified.  Real  cost  growth  is  consistently  measured  in  terms  of  the  cost  of  programs  to  the 
economy  as  a  whole,  not  in  terms  of  the  physical  resources  used  by  the  program.  The 
current  practice  of  using  the  best  available  information  to  prepare  then-year  dollar  estimates 
means  that  program-specific  input  price  increases  that  are  expected  to  exceed  general 
inflation  are  built  into  the  baseline  and  do  not  count  as  cost  growth.  Unanticipated  increases 
in  input  prices  do  contribute  to  measured  cost  growth  and  can  contribute  to  Nunn-McCurdy 
breaches. 

Concluding  Observations  and  Suggestions 
Observations 

•  There  is  no  single  price  or  inflation  index  that  should  be  used  for  all  purposes. 
The  appropriate  index  depends  on  the  mix  of  goods  and  services  under 
consideration.  If  the  context  is  measuring  cost  to  the  economy,  a  broad-index, 
like  the  GDP  deflator,  is  appropriate.  If  the  context  is  narrower,  like  predicting 
the  cost  of  specific  kinds  of  purchases,  a  more  focused  index  is  appropriate. 

•  The  GDP  deflator  and  the  price  indexes  for  particular  sectors  developed  by 
the  BEA  and  BLS  are  based  on  output  prices.  Although  the  DoD’s  purchases, 
including  MDAPs,  are  outputs  from  the  private  sector,  the  cost-based  nature 
of  contract  development  supports  the  use  of  input-price-based  indexes  for 
MDAPs. 
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•  Current  DoD  practices  regarding  the  treatment  of  inflation  support  the  DoD’s 
needs  for  accurate  budgeting  and  for  calculating  real  program  cost  growth. 

•  Although  the  use  of  program-specific  estimates  of  future  input-price  changes 
is  the  best  way  to  ensure  accurate  budgeting  for  MDAPs,  the  estimates 
require  systematic  review  at  both  the  Service  and  OSD  levels  to  resist  a 
possible  tendency  to  accumulate  budget  reserves  in  the  guise  of  preparing 
for  inflation. 

•  Guidance  by  the  OUSD(C)  on  the  use  of  its  indexes  to  determine  budgetary 
requirements  and  develop  program  cost  estimates  currently  calls  for  budgets 
that  (a)  reflect  most  likely  or  full  costs,  and  (b)  use  OUSD(C)  indexes  to 
determine  price  escalation.  The  guidance  further  states  that  the  Comptroller’s 
price  indexes  should  be  used  to  “determine  the  amount  of  price  escalation  for 
a  procurement  line  item,  major  RDT&E  system,  or  construction  item  over  a 
given  time  period”  (DoD,  n.d.).  This  guidance  is  being  revised  to  make  it  clear 
that  most  likely  or  expected  full  costs  in  then-year  dollars  should  be  used  in 
budget  preparation — even  if  this  implies  price  increases  different  from  those 
implied  by  Comptroller’s  indexes — and  that  Comptroller  indexes  must  be 
used  to  convert  then-year  dollar  values  to  constant-dollar  values. 

•  The  use  of  the  GDP  deflator  to  measure  price  increases  for  all  DoD 
procurement  programs  is  conceptually  inappropriate.  Healthcare,  fuel  and 
personnel  have  price  indexes  specific  to  them.  This  is  not  true  for 
procurement.  Empirically,  the  GDP  deflator  may  be  a  reasonable  proxy  for 
procurement  inflation  overall,  though  this  cannot  be  demonstrated.  But  it  does 
not  allow  the  DoD  to  capture  differences  between,  for  example,  ships, 
aircraft,  and  vehicles.  Individual  organizations  often  develop  their  own 
approaches. 

•  This  initial  study  does  not  indicate  what  alternative  system-  or  category- 
specific  indexes  would  provide  better  estimates  of  inflation  for  procuring  the 
various  types  of  systems.  Government  statistical  organizations  do  not  publish 
price  indexes  based  on  the  prices  of  inputs  to  the  production  of  systems,  but 
they  presumably  could. 

•  Current  practice  does  not  appear  consistent  with  either  of  the  notions  of 
constant  prices  noted  at  the  start  of  the  paper.  By  using  tailored  indexes  for 
civilian  personnel,  military  personnel,  fuel,  and  medical  care,  it  does  not 
consistently  calculate  constant  dollar  costs  in  terms  of  resources  foregone  by 
the  economy  as  a  whole.  By  using  the  GDP  deflator  for  procurement,  it  does 
not  consistently  calculate  constant  dollar  costs  in  terms  of  the  value  of  the 
resources  acquired  to  the  DoD. 

•  Some  procurement  price  indexes,  particularly  the  BEA  national  defense 
indexes  for  aviation  and  missiles,  appear  surprisingly  low,  with  negligible 
growth  since  1985.  This  may  be  due,  at  least  in  part,  to  the  way  that  quality 
adjustments  are  identified  and  estimated. 

•  There  has  been  little  systematic  tendency  to  either  overestimate  or 
underestimate  inflation.  Prediction  of  inflation  five  years  in  the  future  has 
been  wrong  by  only  about  0.8%  on  average. 
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Suggestions 

•  Complete  the  planned  revision  of  OUSD(C)  guidance. 

•  Investigate  the  feasibility  of  developing  procurement  price  indexes  tailored  to 
different  kinds  of  equipment.  This  would  involve  deeper  analysis  of  the  BEA 
and  BLS  for  military  systems,  especially  the  use  of  indexes  based  on  the 
prices  of  inputs  to  military  systems. 

•  Compare  the  accuracy  of  inflation  predictions  promulgated  by  OMB  and 
those  developed  by  Global  Insight. 
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Abstract 

Despite  the  fast-growing  interest  in  the  research  of  political  connections  of  either  private- 
sector  firms  or  states,  most  of  the  papers  belong  to  the  economics  or  public  administration 
fields.  There  are  few  studies,  if  any,  that  look  into  the  role  of  firms’  political  connections  in  the 
defense  acquisition  area.  This  paper  makes  an  effort  to  bridge  this  gap  by  investigating  the 
impact  of  political  connections  on  the  excessive  profitability  of  defense  contractors. 

Wang  and  San  Miguel  (2012)  documented  that  defense  contractors  earn  excessive  profits 
relative  to  their  industry  counterparts.  This  study  extends  Wang  and  San  Miguel  (2012)  and 
examines  whether  defense  contractors’  political  connections  (as  measured  by  the  prior 
employment  histories  of  the  board  directors)  influence  contractors’  excessive  profitability.  We 
find  that,  in  contrast  to  the  prediction  of  “corruption  hypothesis,”  the  excessive  profits  are  less 
(more)  pronounced  for  those  contractors  with  politically  connected  (non-connected)  boards. 
This  casts  doubt  on  the  preconceived  notion  that  those  politically  connected  board  members 
are  corrupt  in  nature;  rather,  our  findings  suggest  that  they  may  use  their  experience  to  serve 
a  benevolent  role  to  the  public  in  keeping  defense  contractors  from  opportunistic  profit- 
seeking  behaviors  that  could  reach  or  even  cross  the  federal  government’s  regulatory  redline. 

Introduction 

Political  connections2  of  either  private-sector  firms  or  public  states  has  increasingly 
become  a  popular  research  topic  among  economists,  business  and  public  administration 
scholars,  and  political  scientists.  For  example,  in  regard  to  states’  political  connection  as 
measured  by  representation  in  the  U.S.  Congress,  scholars  have  documented  that  per 
capita  federal  expenditures  at  the  state  level  are  positively  related  to  per  capita  Senate 
representation,  which  gives  rise  to  a  small  state  advantage  (Atlas,  Gilligan,  Hendershott,  & 
Zupan,  1995).  No  similar  advantage  is  found  if  data  is  restricted  to  earmarks  secured  in 
House  appropriations  bills3  (Hoover  &  Pecorino,  2005;  Knight,  2008).  This  seems  to  suggest 
that  political  connection  does  matter  from  a  state’s  perspective. 

Naturally,  a  similar  research  question  exists  for  private-sector  firms:  that  is,  do 
politically  connected  private-sector  firms  derive  economic  benefits  from  such  a  relation? 

Most  studies  intended  to  answer  this  question  somewhat  support  this  conjecture.  For 
instance,  Goldman,  Rocholl,  and  So  (2009)  demonstrated  that  the  market  responds 
positively  (i.e.,  a  positive  abnormal  stock  return  is  observed)  to  the  announcement  of  the 


1  JEL  Classifications:  G38,  H57,  M48. 

2  There  is  no  consensus  regarding  the  definition  of  political  connection.  Definitions  vary  with  specific 
studies. 

3  Note  that  each  state  has  two  senators,  regardless  of  the  population  of  the  state.  The  representation 
in  the  U.S.  House,  however,  is  based  on  state  population. 
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nomination  of  a  board  member  who  is  politically  connected  from  his  or  her  prior  employment 
history  in  the  federal  government,  military  services,  or  as  a  former  representative  of  the  U.S. 
Congress.  Duchin  and  Sosyura  (in  press)  investigated  application  data  for  Troubled  Asset 
Relief  Program  (TARP)  funds  and  found  that  those  firm  applicants  with  political  connections4 
were  more  likely  to  be  funded.  Correia  (2012)  found  that  for  firms  with  irregular  accounting 
practices,  those  with  political  connections  are  less  likely  to  become  the  target  of  Securities 
and  Exchange  Commission  (SEC)  investigation,  and  if  they  are  indeed  investigated,  they 
face  lower  penalties  on  average  than  non-connected  firms.  Khwaja  and  Mian  (2005)  used 
Pakistan  banks’  corporate  lending  data  to  show  the  rent-seeking  behavior  of  politically 
connected  firms.  In  particular,  they  found  that  “political  firms  borrow  45  percent  more  and 
have  50  percent  higher  default  rates.  Such  preferential  treatment  occurs  exclusively  in 
government  banks — private  banks  provide  no  political  favors”  (p.  1371).  It  is  also  worth 
mentioning  that  these  studies  not  only  document  the  real  impacts  of  political  connections, 
but  they  also  share  a  common  theme  suggesting  that  political  connections  are  a  source  of 
corruption  and  underlie  various  rent-seeking  behaviors.  Simply  put,  political  connections 
matter  in  a  negative  way. 

Despite  the  fast-growing  interest  in  the  research  of  political  connections,  most  of  the 
papers  belong  to  the  economics,  political  science,  or  public  administration  field.  There  are 
few  studies,  if  any,  that  look  into  the  role  of  firms’  political  connection  in  the  defense 
acquisition  area,  which  provides  another  proof  of  the  alleged  disciplinary  disconnect5  that 
has  existed  for  a  long  time. 

The  objective  of  this  paper  is  twofold.  First,  we  attempt  to  bridge  the  gap  that  exists 
between  defense  acquisition  study  and  other  relevant  research  fields,  such  as  economics 
and  public  administration.  As  observed  by  many  academicians  and  practitioners,  such  a 
disengagement  of  defense  acquisition  research  (with  other  fields)  is  both  unfortunate  and 
unjustified.  The  society  will  be  better  served  if  such  a  disconnection  is  mitigated.  Toward  this 
goal,  we  build  on  the  extant  literature  and  aim  to  investigate  the  impact  of  political 
connections  (an  established  concept  in  non-defense  research)  on  a  very  important  topic  in 
defense  acquisition,  that  is,  the  excessive  profitability  of  defense  contractors.  Specifically, 
Wang  and  San  Miguel  (2012)  documented  that  defense  contractors  earn  excessive  profits 
relative  to  their  industry  counterparts.  This  study  extends  Wang  and  San  Miguel  (2012)  and 
examines  whether  defense  contractors’  political  connections  (as  measured  by  the  prior 
employment  histories  of  the  board  directors)  influence  contractors’  excessive  profitability. 

Our  second  goal  is  to  test  the  “corruption  hypothesis  of  political  connections”  that  has 
been  suggested  by  existing  literature  in  a  very  particular  and  essential  setting,  that  is,  the 
nation’s  biggest  defense  contractors’  excessive  profitability.  If  the  results  support  the 


4  The  definition  of  political  connection  in  Duchin  and  Sosyura  (2012)  takes  several  forms,  including 
lobbying,  campaign  contributions,  and  employment  history  of  directors. 

5  Such  disconnect  exists  between  public  administration  and  military  administration  (Albano,  Snider,  & 
Thai,  2012),  and  more  generally,  between  economics  and  military-related  research  (Rogerson,  1994). 
Rogerson  (1994)  stated,  “Defense  procurement  is  unique  among  regulated  industries  in  the  United 
States  in  that  economists  have  played  virtually  no  role  in  helping  shape  its  regulatory  practices  and 
institutions.  Perhaps  this  is  due  to  the  barrier  to  entry  created  by  the  need  to  first  learn  about 
procurement  practices  or  to  a  lingering  distaste  for  military  matters  among  academics.  Whatever  the 
reason,  this  lack  of  economic  input  is  unfortunate,  because  many  of  the  regulatory  and  policy  issues 
in  defense  procurement  involve  the  types  of  incentive  issues  that  economists  are  very  good  at 
analyzing.  My  own  hope  is  that  economists  are  on  their  way  to  colonizing  this  new  policy  frontier  and 
that  some  of  the  ideas  discussed  in  this  article  will  play  a  role  in  shaping  policy  debates  over  the  next 
decade”  (p.  87). 
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corruption  story,  then  political  connections  would  become  a  very  serious  concern  of  policy¬ 
makers  because  defense  spending  is  a  substantive  portion  of  government  expenditures.  On 
the  other  hand,  if  such  a  conjecture  is  not  grounded,  what  are  the  findings  and  what  is  the 
explanation? 

The  remainder  of  the  paper  is  organized  as  follows.  The  section  titled  Sample 
describes  our  sample.  The  section  titled  Measuring  Political  Connections  and  Hypotheses 
Development  introduces  the  measure  of  political  connections,  followed  by  the  development 
of  hypotheses  on  the  relationship  between  excessive  profitability  and  political  connections, 
based  on  extant  literature  and  observations.  Empirical  results  and  findings  are  in  the  section 
titled  Empirical  Results  and  Findings.  The  final  section  concludes  the  paper. 

Sample 

We  start  with  the  same  sample  used  in  Wang  and  San  Miguel  (2012).  Specifically, 
they  use  fedspending.org  as  the  data  source  to  identify  the  top  500  recipients  of  defense 
contracts  for  2008.  Out  of  these  top  500  firms,  1 1 2  are  traded  on  public  stock  exchanges. 
These  1 1 2  public  firms  became  the  main  sample  of  their  analyses.  Our  sample  is  a  reduced 
version  of  Wang  and  San  Miguel  (2012)  in  that  we  delete  16  firms  that  are  missing  from  the 
Corporate  Library  database,  which  we  use  to  identify  the  political  connections  of  each  firm’s 
board  members.  Table  1  lists  the  name,  dollar  awarded,  rank,  stock  ticker,  SIC  code,  and 
public  stock  exchange  code  for  these  96  public  firms. 

Table  1.  Firms  in  The  Main  Sample:  96  Public  U.S.  Firms  From  the  2008 

Top  500  list 


Company  Name 

LOCKHEED  MARTIN  CORP 
NORTHROP  GRUMMAN  CORP. 

BOEING  CO. 

RAYTHEON  CO. 

GENERAL  DYNAMICS  CORP. 

UNITED  TECHNOLOGIES  CORP. 

L-3  COMMUNICATIONS  HOLDINGS 
KBR  INC. 

NAVISTAR  INTERNATIONAL  CORPORATION 
ITT  CORPORATION 
SCIENCE  APPLICATIONS  INTL  CORP 
GENERAL  ELECTRIC  COMPANY 
COMPUTER  SCIENCES  CORP. 

HUMANA,  INC. 

TEXTRON,  INC. 

HEALTH  NET,  INC 
URS  CORP. 

HEWLETT-PACKARD  CO. 


Contracted_dollars_2008 

Rank 

$29,363,894,334 

1 

$23,436,442,251 

2 

$21,838,400,709 

3 

$13,593,610,345 

6 

$13,490,652,077 

7 

$8,283,275,612 

8 

$6,675,712,135 

9 

$5,997,147,425 

10 

$4,761,740,206 

11 

$4,355,423,578 

13 

$3,885,932,047 

14 

$3,518,136,891 

15 

$3,230,197,590 

16 

$2,952,008,623 

18 

$2,827,900,303 

19 

$2,438,349,117 

21 

$2,402,033,979 

22 

$1,938,638,634 

26 

Stock 

Ticker 

SIC 

EXCHG 

(11=NYSE, 

12=AMEX, 

14=NASDAQ) 

LMT 

3760 

11 

NOC 

3812 

11 

BA 

3721 

11 

RTN 

3812 

11 

GD 

3790 

11 

UTX 

3720 

11 

LLL 

3663 

11 

KBR 

1623 

11 

NAV 

3711 

11 

ITT 

3812 

11 

SAI 

7373 

11 

GE 

9997 

11 

CSC 

7370 

11 

HUM 

6324 

11 

TXT 

3721 

11 

HNT 

6324 

11 

URS 

8711 

11 

HPQ 

3570 

11 
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ALLIANT  TECHSYSTEMS,  INC. 

$1,928,045,694 

27 

ATK 

3480 

11 

OSHKOSH  TRUCK  CORP. 

$1,863,726,822 

30 

OSK 

3711 

11 

HARRIS  CORP. 

$1,841,470,263 

31 

HRS 

3663 

11 

HONEYWELL,  INC. 

$1,721,547,997 

33 

HON 

3728 

11 

FORCE  PROTECTION  INDUSTRIES,  (INC) 

$1,360,427,189 

36 

FRPT 

3790 

14 

CACI  INTERNATIONAL  INC 

$1,324,104,004 

37 

CACI 

7373 

11 

AMERISOURCE  BERGEN  CORP 

$1,298,059,841 

38 

ABC 

5122 

11 

ROCKWELL  COLLINS 

$1,290,813,364 

39 

COL 

3728 

11 

SHAW  GROUP,  INC. 

$1,162,267,243 

40 

SHAW 

8711 

11 

VALERO  ENERGY  CORPORATION 

$1,043,869,551 

43 

VLO 

2911 

11 

JACOBS  ENGINEERING  GROUP  INC 

$951,295,410 

45 

JEC 

1600 

11 

VSE  CORP. 

$910,970,473 

47 

VSEC 

8711 

14 

MCKESSON  CORPORATION 

$903,799,326 

48 

MCK 

5122 

11 

CARDINAL  HEALTH  INC 

$856,333,988 

50 

CAH 

5122 

11 

DELL  COMPUTER  CORPORATION 

$852,813,703 

51 

DELL 

3571 

14 

EXXON  MOBIL  CORP. 

$836,548,150 

52 

XOM 

2911 

11 

MANTECH  INTERNATIONAL  CORP 

$655,579,972 

61 

MANT 

7373 

14 

FUR  SYSTEMS,  INC 

$507,944,847 

71 

FUR 

3812 

14 

GOODRICH  CORPORATION 

$487,753,671 

73 

GR 

3728 

11 

TETRA  TECH,  INC. 

$472,960,770 

77 

TTEK 

8711 

14 

IBM  CORP. 

$438,446,918 

81 

IBM 

7370 

11 

PERINI  CORP. 

$436,363,793 

82 

TPC 

1540 

11 

FLUOR  CORP. 

$430,878,065 

84 

FLR 

1600 

11 

CERADYNE  INC 

$417,616,849 

86 

CRDN 

3290 

14 

AECOM  TECHNOLOGY  CORPORATION 

$380,250,228 

91 

ACM 

8711 

11 

AT&T  INC. 

$371,099,463 

95 

T 

4813 

11 

KRAFT  FOODS  INC 

$367,840,952 

97 

KFT 

2000 

11 

OWENS  &  MINOR  INC 

$365,861,498 

99 

OMI 

5047 

11 

CUBIC  CORP. 

$354,623,567 

102 

CUB 

3812 

11 

GREAT  LAKES  DREDGE  &  DOCK 

CORPORATION 

$324,475,211 

113 

GLDD 

1600 

14 

CATERPILLAR,  INC. 

$323,676,276 

114 

CAT 

3531 

11 

PROCTER  &  GAMBLE  CO. 

$321,983,149 

115 

PG 

2840 

11 

TYSON  FOODS  INC 

$319,486,334 

117 

TSN 

2011 

11 

VERIZON  COMMUNICATIONS 

$319,365,283 

118 

VZ 

4812 

11 

CHEVRONTEXACO  CORPORATION 

$310,558,853 

122 

CVX 

2911 

11 

SRA  INTERNATIONAL,  INC. 

$297,913,799 

128 

SRX 

7370 

11 

GRANITE  CONSTRUCTION  CO. 

$292,263,100 

131 

GVA 

1600 

11 

ACCENTURE 

$288,517,607 

132 

ACN 

8742 

11 

JOHNSON  CONTROLS,  INC. 

$285,123,825 

134 

JCI 

2531 

11 

EXPRESS  SCRIPTS 

$215,750,049 

162 

ESRX 

6411 

14 
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CONOCOPHILUPS 

$206,348,789 

167 

COP 

2911 

11 

TYCO  INTERNATIONAL  LTD 

$202,567,751 

172 

TYC 

9997 

11 

COMTECH  TELECOMMUNICATIONS  CORP. 

$202,082,670 

173 

CMTL 

3663 

14 

GENERAL  MILLS,  INC. 

$200,017,932 

176 

GIS 

2040 

11 

TESORO  HAWAII  CORPORATION 

$199,447,230 

177 

TSO 

2911 

11 

AEROVIRONMENT  INC 

$192,462,098 

182 

AVAV 

3721 

14 

AAR  CORP. 

$187,717,969 

187 

AIR 

5080 

11 

SYSCO  CORPORATION 

$179,074,006 

195 

SYY 

5140 

11 

REFINERY  HOLDING  COMPANY  L  P 

$177,749,226 

198 

WNR 

2911 

11 

DEERE  &  CO. 

$164,340,456 

206 

DE 

3523 

11 

VIASAT,  INC 

$156,815,300 

217 

VSAT 

3663 

14 

ORBITAL  SCIENCES  CORP. 

$153,884,356 

223 

ORB 

3760 

11 

PEPSICO  INC 

$149,527,183 

231 

PEP 

2080 

11 

UNISYS 

$142,990,124 

239 

UIS 

7373 

11 

BALL  CORP 

$131,696,095 

259 

BLL 

3411 

11 

CONAGRA,  INC. 

$125,264,234 

270 

CAG 

2000 

11 

ORACLE  CORP. 

$122,646,803 

274 

ORCL 

7372 

14 

GENERAL  MOTORS  CORP. 

$120,929,817 

279 

GM 

3711 

11 

EATON  CORP. 

$117,792,917 

286 

ETN 

3620 

11 

UNILEVER  NV 

$112,089,508 

292 

UL 

2000 

11 

MOOG,  INC. 

$111,608,841 

293 

MOG.A 

3728 

11 

ALON  USA  L.P. 

$111,102,800 

296 

ALJ 

2911 

11 

COCA-COLA  ENTERPRISES  INC 

$93,991,833 

343 

CCE 

2086 

11 

XEROX  CORP. 

$91,275,424 

356 

XRX 

3577 

11 

JOHNSON  &  JOHNSON 

$89,990,235 

363 

JNJ 

2834 

11 

CAMPBELL  SOUP  CO. 

$88,645,010 

367 

CPB 

2030 

11 

INTERMEC  CORPORATION 

$83,566,808 

388 

IN 

3577 

11 

CAE  CORP 

$83,563,697 

389 

CAE 

3690 

11 

DEL  MONTE  FOODS  COMPANY 

$77,962,809 

419 

DLM 

2000 

11 

AMERICAN  SCIENCE  AND  ENGRG 

$76,545,302 

429 

ASEI 

3844 

14 

MICHAEL  BAKER  CORP. 

$74,263,592 

437 

BKR 

8711 

12 

KIMBERLY-CLARK  CORP. 

$69,832,351 

454 

KMB 

2621 

11 

ESTERLINE  TECHNOLOGIES  CORP 

$68,716,933 

462 

ESL 

3823 

11 

INTEGRAL  SYSTEMS,  INC. 

$67,261,245 

473 

ISYS 

7373 

14 

MINE  SAFETY  APPLIANCES  CO. 

$67,166,647 

474 

MSA 

3842 

11 

WORLD  FUEL  SERVICE  CORP. 

$66,258,375 

478 

INT 

5172 

11 

SARA  LEE  CORPORATION 

$65,361,053 

482 

SLE 

2000 

11 

WILLIAMS  COMPANIES  INC 

$65,024,852 

483 

WMB 

4922 

11 

HORIZON  LINES  LLC 

$65,008,856 

484 

HRZ 

4400 

11 

Tablel  shows  that  most  of  the  firms  in  our  sample  are  listed  on  the  NYSE  or 
NASDAQ,  indicating  that  big  defense  contractors  are  likely  to  be  established  companies.  For 
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each  of  the  96  firms,  we  use  their  stock  ticker  to  map  into  the  Compustat  database  and 
extract  various  accounting  variables  across  a  three-year  range  of  2007-2009.  Note  that  our 
base  year  is  2008.  The  reason  we  include  two  additional  years  of  data  (i.e.,  2007,  one  year 
prior,  and  2009,  one  year  after)  is  to  expand  the  sample  size  and  simultaneously  ensure  that 
the  status  of  the  top  500  defense  contractors  in  2008,  as  well  as  the  political  connections  of 
the  board  members  in  2008,  can  be  assumed  to  be  stationary  and  be  passed  onto  2007  and 
2009  for  the  same  firm,  due  to  a  short  elapse  of  time.  Expanding  our  sample  to  a  three-year 
range  yields  a  total  of  276  firm-years,  with  93  each  for  2007  and  2009  and  90  for  2008. 
Following  Wang  and  San  Miguel  (201 2),  we  denote  the  excessive  profit  of  a  particular  firm- 
year  as  the  difference  between  this  firm-year’s  return  on  assets  (ROA)6  and  the  ROA  of  an 
“industry-year-size”  matched  benchmark  firm  that  is  not  on  the  1 12-firm  list.7 

Table  2  presents  basic  statistics  of  descriptive  accounting  measures  for  the  90 
sample  firms  in  Fiscal  Year  2008. 8  In  particular,  we  report  total  assets,  total  sales  (revenue), 
dollar  awarded  as  percentage  of  revenue,  and  excessive  profit  as  measured  by  the  matched 
ROA.  The  mean  values  of  total  assets  and  total  revenue  were  $35  billion  and  $33  billion, 
respectively.  The  government  contracts  contributed  about  18%  of  these  firms’  2008  revenue 
on  average.9  Overall,  these  firms  earned  an  excessive  ROA  of  3%,  which  is  statistically 
significant  at  a  5%  significance  level,  confirming  Wang  and  San  Miguel’s  (2012)  findings  that 
top  defense  contractors  receive  excessive  profits  relative  to  their  industry  peers. 


6  To  keep  the  paper  concise,  we  exclusively  use  ROA  as  the  profitability  metric  in  this  study.  Other 
alternative  profit  measures  yield  similar  results. 

7  “The  benchmark  firm-year  is  selected  based  on  a  three-dimension  match  on  industry,  year  and  size. 
Specifically,  we  go  to  the  same  industry-year  where  industry  membership  is  defined  as  four-digit  SIC 
codes,  and  identify  the  non-defense  (i.e.,  not  on  our  1 12-firm  list)  firm  that  has  the  best  size  match 
with  our  defense  firm-year.  The  difference  between  the  profit  of  the  firm-year  investigated  and  the 
profit  of  the  benchmark  firm-year  will  be  the  measure  of  ‘excessive  profit’”  (Wang  &  San  Miguel,  2012, 
p.  397). 

8  We  lost  six  firms  for  Year  2008  due  to  missing  data  from  Compustat. 

9  A  concern  that  has  been  raised  here  is  that  a  significant  portion  of  our  sample  firms  may  have  much 
lower  than  18%  of  their  total  revenue  that  is  attributable  to  DoD  contracts,  and  hence,  are  not  really 
“defense  contractors”  as  the  term  is  generally  understood.  Consequently,  if  Sara  Lee  had  only  1%  of 
2008  sales  from  defense  contracts,  one  cannot  attribute  much,  if  any,  of  Sara  Lee’s  excessive  profits 
to  its  defense  contracts.  We  provide  a  few  arguments  to  address  the  aforementioned  concern.  First, 
our  sample  focuses  on  DoD  contractors,  a  much  broader  concept  than  a  few  prominent  major 
weapon  manufacturers.  In  that  regard,  an  average  18%  revenue  from  DoD  is  a  reasonably  decent 
number.  Second,  the  central  metric  of  our  analysis  is  the  excessive  profit,  and  because  profit  is  only  a 
small  portion  of  revenue,  a  relatively  small  percentage  of  DoD  revenue  could  have  a  much  larger 
impact  on  profit  if  firms  do  derive  larger  profits  from  DoD  contracts  than  they  can  generate  from  their 
non-DoD  business.  Third,  it  is  worth  mentioning  that  the  specific  concern  as  expressed  by  using  the 
Sara  Lee  example  above  is  already  addressed,  if  not  completely  removed,  by  our  definition  of  the 
three-way  industry-year-size  matched  excessive  profit  measure.  In  particular,  if  Sara  Lee  had  a  super 
good  year  for  whatever  reason  that  is  non-DoD  related,  we  expect  that  its  benchmark  firm,  i.e.,  the 
firm  that  is  in  the  same  industry  and  has  similar  size  (but  without  federal  contracts),  would  also  be 
impacted  in  a  similar  way  and  display  a  superior  profit  likewise  in  the  same  year.  Hence,  the 
excessive  profit  of  Sara  Lee,  which  is  the  difference  between  Sara  Lee’s  profit  and  its  benchmark 
firm’s  profit,  would  be  only  attributable  to  the  fact  that  Sara  Lee  has  DoD  contracts  while  its 
benchmark  firm  has  not.  Last  but  not  least,  despite  that  we  believe  our  current  full-sample  approach 
is  sound,  we  nevertheless  proceed  to  perform  a  robustness  analysis,  which  only  includes  the 
subsample  that  consists  of  only  those  firms  with  at  least  25%  of  total  revenue  generated  from  DoD 
contracts.  Untabulated  results  show  that  all  our  findings  are  intact. 
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Table  2.  The  Basic  Statistics  of  90  Sample  Firms  in  Year  2008 


Mean 

Median 

Min 

Max 

Std  Dev 

Total  Assets  (millions) 

34,962 

7,242 

147 

797,769 

94,895 

Total  Sales  (millions) 

32,656 

12,542 

160 

425,071 

59,570 

Dollar  Awarded  as  Percent 

17.56 

6 

0.06 

103.00 

22.79 

of  Sales  (%) 

Excessive  ROA 

0.03 

0.02 

-0.18 

0.32 

0.10 

Measuring  Political  Connections  and  Hypotheses  Development 


Measuring  Political  Connections 

There  is  no  unanimously  agreed-upon  definition  of  the  term  political  connection ,10 
Scholars  have  used  various  forms  of  concepts  in  different  research  settings.  For  example, 
Mara  Faccio,  in  a  series  of  solo  and  coauthored  papers,* 11  defined  a  firm’s  political 
connection  as  follows:  “A  company  is  defined  as  being  connected  with  a  politician  if  at  least 
one  of  its  largest  shareholders  (anyone  controlling  at  least  10  percent  of  voting  shares)  or 
one  of  its  top  officers  (CEO,  president,  vice-president,  chairman,  or  secretary)  is  a  member 
of  parliament,  a  minister,  or  is  closely  related  to  a  top  politician  or  party”  (Faccio,  2006,  p. 
369).  This  definition  by  Faccio  is  not  appropriate  for  any  U.S. -based  study  because  U.S. 
regulations  pretty  much  rule  out  the  possibility  of  anybody  simultaneously  serving  a  high- 
rank  public  service  role  and  a  top  executive  role  in  a  private-sector  firm.  In  the  United  States, 
if  a  present  executive  of  a  private-sector  firm  is  appointed  as  a  high-rank  government 
official,  he  or  she  must  quit  his  or  her  current  job.  As  a  testimony  of  this  fact,  Faccio  (2010) 
found  that  under  her  definition,  only  13  out  of  the  6,007  U.S.  firms  in  the  Worldscope 
database  can  be  labeled  as  “politically  connected  firms.”  In  short,  this  first  definition  applies 
more  to  international  countries,  such  as  Indonesia,  Malaysia,  or  Italy. 

The  second  definition  of  political  connection  focuses  on  campaign  contributions  and 
lobbying  activities.  For  instance,  Correia  (2012)  found  that  firms’  political  connections 
established  by  contributions  to  congressmen  and  by  lobbying  the  SEC  reduce  those  firms’ 
enforcement  costs  by  the  SEC.  Specifically,  those  firms  are  less  likely  to  be  investigated  by 
the  SEC,  and  even  if  they  are  investigated,  the  average  penalty  is  lower  for  them.  Other 
studies  that  adopted  this  definition  include  Roberts  (1990),  Kroszner  and  Stratmann  (1998), 
and  Ang  and  Boyer  (2000).  The  problem  with  this  definition  is  the  low  explanatory  power. 

For  instance,  Goldman  et  al.  (2009)  found  that  controlling  industry  effect  significantly 
reduces  the  explanatory  power  of  campaign  donation.  Moreover,  Jayachandran  (2006) 
questioned  the  causal  effect  of  firms’  donations  on  firm  value.  To  recap,  the  second 
definition,  based  on  campaign  donation  or  lobbying  expenditure,  at  most  provides  a  noisy 
measure  of  political  connection. 

The  third  alternative  definition  of  political  connection  is  derived  from  board  directors’ 
prior  employment  history  in  the  federal  government,  including  in  the  legislative,  executive, 
and  judiciary  branches,  and  in  the  military  Services.  Since  in  the  U.S.,  congressmen, 
government  executives,  and  military  generals  are  allowed  to  serve  on  the  boards  of  private- 
sector  firms  after  their  retirement  from  public  service  (and  they  frequently  do  so),  firms’ 


10  From  this  point  on,  we  restrict  our  attention  on  political  connections  to  private-sector  firms  rather 
than  public  states.  One  example  of  a  public  state’s  political  connection  was  introduced  previously. 

11  See  Faccio  (2006),  Faccio  (2010),  Faccio,  Masulis,  and  McConnell  (2006),  and  Chaney,  Faccio, 
and  Parsley  (201 1). 
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political  connections  through  board  members  receive  substantial  attention.  Many  U.S. -based 
studies  follow  the  suit  of  this  particular  definition.  To  name  a  few,  Agrawal  and  Knoeber 
(2001)  found  that  firms  for  which  politics  plays  a  more  important  role  tend  to  be  more 
“politically  connected”  (i.e. ,  they  tend  to  have  more  politically  experienced  directors  on  their 
boards).  Goldman  et  al.  (2009)  showed  the  market  value  relevance  of  the  addition  of  a 
newly  appointed,  politically  connected  board  member.  Moreover,  they  differentiate  between 
political  connections  to  the  Republican  versus  Democratic  parties  and  provide  evidence  that 
the  market  values  of  these  two  different  types  of  politically  connected  firms  responded 
differently  to  George  W.  Bush’s  2000  presidential  win. 

Since  our  sample  is  strictly  U.S.  based,  it  is  natural  to  follow  the  third  definition  of 
political  connection.  Specifically,  we  use  the  2008  Directorships  database  that  is  provided  by 
Corporate  Library  LLC.  In  this  annual  directorship  dataset,  Corporate  Library  records  each 
individual  director’s  information  through  compiling  data  from  firms’  publicly  disclosed  proxy 
statements.  One  key  field  in  this  database  is  a  director’s  biography,  including  detailed 
employment  history.  We  use  a  series  of  keywords  to  search  each  individual  director’s 
biography  statement  and  identify  whether  this  particular  director  is  politically  connected.  The 
keywords  we  use  are  comprehensive  to  ensure  a  maximum  catch  of  politically  connected 
directors.  The  complete  list  of  our  search  keywords  follows:  senator,  congressman, 
congresswoman,  congress,  representative,  federal,  secretary,  admiral,  general,  army,  navy, 
air  force,  department  of  defense,  DoD,  commissioner,  ambassador,  administrator,  attorney 
general,  governor,  director,  council. 

We  apply  this  keyword  search  to  the  biography  statement  as  of  Year  2008  for  each 
director  who  sits  on  the  board  of  any  of  our  96  sample  firms.  Once  we  find  a  “hit”  of  a 
keyword,  we  read  the  biography  and  make  sure  this  particular  director  is  correctly  flagged  as 
one  who  is  politically  connected.12  At  Year  2008,  our  96  sample  firms  have  989  directors  in 
total,  indicating  an  average  board  size  of  10.3  directors.  Out  of  these  989  directors,  923  are 
unique  individuals,  of  which  157  are  identified  as  politically  connected  directors.  Put  simply, 
17%  of  the  directors  have  prior  employment  history  with  the  federal  government  or  military 
Services.  The  data  also  indicate  that  77  out  of  96  firms  have  at  least  one  politically 
connected  director  on  their  board;  that  is,  80%  of  our  top  defense  contractors  have  some 
degree  of  political  connection  through  the  board  of  directors.  To  get  a  benchmark  sense,  it  is 
worth  mentioning  that  Goldman  et  al.  (2009),  using  a  very  similar  definition  of  political 
connection  as  our  study,  documented  that  at  Year  2000,  153  of  the  S&P  500  companies 
(i.e.,  31%)  had  at  least  one  board  member  with  a  political  connection.  Therefore,  the  main 
message  is  that  top  defense  contractors  are  much  more  likely  to  have  a  politically  connected 
board  than  non-contractor  firms. 


12  An  example  of  a  politically  connected  director’s  profile  is  General  John  M.  Shalikashvili,  who  served 
as  a  board  director  of  L-3  Communications  Holdings,  Inc.,  at  Year  2008.  The  following  excerpt  was 
from  the  company’s  proxy  statement:  “General  John  M.  Shalikashvili,  director  since  August  1998  and 
member  of  the  Compensation  and  Nominating/Corporate  Governance  Committees.  General 
Shalikashvili  (U.S.  Army — Ret.)  is  an  independent  consultant  and  a  Visiting  Professor  at  Stanford 
University.  General  Shalikashvili  was  the  senior  officer  of  the  United  States  military  and  principal 
military  advisor  to  the  President  of  the  United  States,  the  Secretary  of  Defense  and  the  National 
Security  Council  when  he  served  as  the  thirteenth  Chairman  of  the  Joint  Chiefs  of  Staff,  Department 
of  Defense,  for  two  terms  from  1 993  to  1 997.  Prior  to  his  tenure  as  Chairman  of  the  Joint  Chiefs  of 
Staff,  he  served  as  the  Commander  in  Chief  of  all  United  States  forces  in  Europe  and  as  NATO’s 
tenth  Supreme  Allied  Commander,  Europe  (SACEUR).  He  has  also  served  in  a  variety  of  command 
and  staff  positions  in  the  continental  United  States,  Alaska,  Belgium,  Germany,  Italy,  Korea,  Turkey 
and  Vietnam.” 
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Hypotheses  Development 

In  this  subsection,  we  derive  alternative  hypotheses  on  the  relationship  between 
defense  contractors’  excessive  profitability  and  their  political  connections,  based  on  extant 
literature  and  observations.  Most  of  the  prior  literature  suggests  the  “corruption”  role  of 
political  connection  (i.e.,  the  firms  with  political  connections  opportunistically  take  advantage 
of  this  favorable  relation  and  inappropriately  derive  private  benefits  for  the  firm  at  the 
sacrifice  of  social  welfare).  For  example,  Duchin  and  Sosyura  (in  press)  found  that  politically 
connected  firms  are  more  likely  to  get  TARP  funds,  yet  their  performance  was  inferior  to  that 
of  unconnected  firms.  This  clearly  indicates  that  political  connection  is  a  source  of 
“corruption”  and  “inefficiency.”  Correia  (2012)  presented  evidence  showing  that  firms  use 
their  political  influence  to  avoid  the  scrutiny  of  the  SEC  or  mitigate  the  punitive  damage  in 
the  case  of  financial  reporting  irregularity.  Faccio  et  al.  (2006)  analyzed  a  unique  dataset 
that  covers  35  countries  during  1997-2002  and  found  that  those  politically  connected  firms 
are  far  more  likely  to  be  bailed  out  during  financial  distress  than  non-connected  firms  in  a 
similar  economic  crisis.  Moreover,  after  bailout,  those  firms  with  political  connections 
significantly  underperform  unconnected  firms.  Chaney  et  al.  (2011)  documented  that 
politically  connected  firms  have  poorer  earnings  quality  than  their  non-connected 
counterparts.  All  of  the  studies  mentioned  previously  collectively  convey  a  consistent 
message:  that  is,  political  connection  is  associated  with  various  rent-seeking  behaviors. 
Applying  this  corruption  proposition  of  political  connections  to  the  defense  contractors’ 
excessive  profit,  we  have  the  following  hypothesis: 

Hypothesis  (H):  The  defense  contractors’  excessive  profitability  is  more 
pronounced  for  those  with  political  connections.  Non-connected  firms  should 
exhibit  a  less  excessive  profit. 

While  this  hypothesis  sounds  like  a  reasonable  conjecture  given  all  evidence  in  the 
extant  literature,  an  alternative  hypothesis  nevertheless  could  exist.  In  particular,  if  defense 
contractors,  a  unique  subset  of  universal  firms,  have  different  and  non-opportunistic  motives 
for  establishing  political  connections,  then  the  story  could  be  very  different.  Given  the  unique 
nature  of  the  defense  procurement  business,  it  is  quite  likely  that  commonality  may  not 
prevail  here.  For  instance,  one  distinctive  feature  of  defense-related  business  is  the 
complexity  of  regulation,  which  often  requires  substantive  professional  and  inside 
knowledge  to  truly  understand.  The  Federal  Acquisition  Regulation  (FAR)  alone  consists  of 
thousands  of  pages  full  of  government-specific  terminologies.  Further,  a  firm  that  is  doing 
business  with  the  Department  of  Defense  (DoD)  is  under  the  scrutiny  of  various  government 
agencies,  such  as  the  Government  Accountability  Office  (GAO),  the  Defense  Contract  Audit 
Agency  (DCAA),  and  others.  There  is  a  high  cost  of  non-compliance.  A  defense  contractor 
that  is  found  to  engage  in  misconduct  could  face  various  penalties  including  settlement  with 
fine,  civil  or  criminal  investigation,  suspension,  or  even  debarment.  If  defense  contractors 
believe  that  these  redlines  are  costly  to  cross,  they  may  have  incentives  to  hire  the  best 
talent  with  professional  and  institutional  knowledge  to  help  them  avoid  such  behavior.  For 
example,  a  March  22,  1991,  article  in  The  Wall  Street  Journal,  titled  “Northrop  Nominates 
Three  for  Its  Board,”  reported  that 

The  nominees  are  Joseph  A.  Califano  Jr.,  59  years  old,  a  Washington 
attorney  and  former  Secretary  of  Health,  Education  and  Welfare  under 
President  Jimmy  Carter;  Jack  Edwards,  62,  a  Washington  lawyer  and 
formerly  the  ranking  Republican  congressman  on  the  Defense  Appropriations 
Subcommittee;  and  retired  Gen.  John  T.  Chain  Jr.,  56,  a  35-year  Air  Force 
veteran  who  this  year  retired  as  commander-in-chief  of  the  Strategic  Air 
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Command  to  become  executive  vice  president  of  operations  of  Burlington 
Northern  Railroad  Co. 

A  company  spokesman  said  in  the  news  announcement,  “[These]  board  members 
are  chosen  for  the  breadth  of  their  experience  and  counsel”  (“Northrop  Nominates,”  1991). 
Moreover,  Kent  Kresa,  then  Northrop  president  and  chief  executive  officer,  further 
commented,  “These  men  bring  to  Northrop  unsurpassed  experience  and  knowledge  in  their 
own  fields,  and  a  diversity  that  will  serve  us  well  as  we  shape  the  company  to  match  the 
changes  taking  place  in  the  country  and  the  world”  (“Northrop  Nominates,”  1991).  Note  that 
two  of  the  individuals  are  attorneys  and  all  three  of  them  had  extensive  and  high-profile 
government  or  military  experiences.  Their  expertise  and  experience,  if  used  under  good 
intention,  would  greatly  help  Northrop  comply  with  the  regulatory  and  executive  rules. 
Recognizing  this  potential  competing  theory,  we  offer  the  following  alternative  hypothesis: 

Alternative  Hypothesis  (AH):  The  defense  contractors’  excessive  profitability 
is  less  pronounced  for  those  with  political  connections.  Non-connected  firms 
should  exhibit  a  more  excessive  profit. 

Both  H  and  AH  have  reasonable  justifications.  Which  one  is  factually  supported?  The 
next  section  empirically  investigates  this  issue. 

Empirical  Results  and  Findings 

Univariate  Analysis 

We  first  report  the  univariate  statistics  of  key  variables.  Recall  from  the  Sample 
section  that  we  have  276  firm-years  in  a  three-year  range  of  2007-2009.  We  classify  each 
of  these  276  firm-years  into  one  of  the  two  mutually  exclusive  groups.  The  first  group, 
labeled  as  “non-politically  connected”  firms,  consists  of  all  firm-years  for  which  none  of  a 
firm’s  Year-2008  board  members  had  political  connection  through  his  or  her  prior 
employment.  All  of  the  other  firm-years  that  are  not  in  the  first  group  had  at  least  one  of  the 
firm’s  board  members  being  classified  as  a  “politically  connected  director”  and  hence  belong 
to  the  second  group  called  “politically  connected”  firms.  Out  of  the  276  firm-years,  54  are 
politically  non-connected  and  222  are  connected. 
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Table  3. 


The  Univariate  Comparison  of  Key  Variables  Between  Politically 
Connected  and  Non-Connected  Firm-Years 


Group 

N 

Variable 

Mean 

Std  Dev 

Politically 

Non- 

Connected 

54 

Total  Assets 

(millions) 

13,535 

23,945 

Total  Sales 

(millions) 

22,754 

30,769 

Dollar  awarded  as 

percent  of  sales 

(%) 

8.52 

11.73 

Excessive  ROA 

0.04 

0.09 

Politically 

Connected 

222 

Total  Assets 

(millions) 

41,339 

103,331 

Total  Sales 

(millions) 

33,060 

56,377 

Dollar  awarded  as 

percent  of  sales 

(%) 

21.59 

28.00 

Excessive  ROA 

0.01 

0.08 

We  have  several  immediate  observations  from  Table  3.  First,  politically  connected 
defense  contracting  firms  are  much  bigger  than  non-connected  ones.  Measured  by  assets 
(revenue),  a  typical  politically  connected  firm  is  three  (one-and-a-half)  times  as  big  as  a 
typical  non-connected  firm.  Second,  defense  contracts  account  for  a  much  bigger  portion  of 
total  revenue  for  politically  connected  contractors  than  for  non-connected  ones.  Specifically, 
about  21.6%  (as  opposed  to  8.5%)  of  total  revenue  is  generated  by  defense  contracts  for 
politically  connected  firms  (as  opposed  to  non-connected  firms).  This  particular  evidence  is 
consistent  with  Agrawal  and  Knoeber  (2001),  who  found  that  for  those  firms  in  which  sales 
to  government  plays  a  more  important  role,  the  presence  of  politically  connected  directors 
on  the  board  is  greater  as  well.  It  is  also  in  line  with  the  finding  of  Goldman,  Rocholl  &  So  (in 
press)  that  political  connections  affect  the  allocation  of  procurement  contracts.  Nevertheless, 
we  would  like  to  stress  that  just  because  there  is  a  positive  association  between  the  political 
connection  and  the  defense  contract  dollar  as  a  percentage  of  revenue  does  not  necessarily 
indicate  a  rent-seeking  or  corruption  story.  It  is  plausible  that  the  hiring  of  political 
experience  is  well  intentioned  and  that  those  valuable  experiences  are  legitimately  used  to 
compete  for  government  contracts  in  a  lawful  and  ethical  way.  Last  but  not  least,  a 
univariate  comparison  on  excessive  profits  (as  measured  by  excessive  ROA)  between 
politically  connected  and  non-connected  groups  demonstrates  that  the  former  displays  a 
much  less  pronounced  excessive  profit  than  the  latter  (4%  versus  1%).  This  suggests  that 
preliminary  evidence  casts  doubt  on  the  corruption  (or  rent-seeking)  hypothesis  and  favors 
our  alternative  hypothesis,  which  supports  the  non-opportunistic  motives  for  establishing 
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political  connections.  That  said,  a  more  sophisticated  approach  (beyond  univariate  analysis) 
is  needed  to  provide  more  convincing  evidence. 

Multivariate  Analysis 

In  this  subsection,  we  use  a  multivariate  regression  method  to  examine  whether  the 
evidence  against  the  corruption  hypothesis  in  a  univariate  context  persists  in  a  multivariate 
setting.  Put  another  way,  we  want  to  inspect  whether  our  preliminary  finding  based  on  a 
univariate  relation  is  robust  to  controlling  all  known  determinants  of  defense  contractors’ 
excessive  profits.  Needless  to  say,  our  dependent  variable  (i.e.,  the  left-hand-side  variable) 
is  the  firms’  excessive  profits,  and  our  main  variable  of  interest  on  the  right-hand  side  is  the 
firms’  political  connections.  To  ensure  that  the  impact  of  political  connection  on  excessive 
profit  is  incremental  to  the  effects  of  all  the  other  known  determinants  of  excessive  profits, 
we  need  to  include  a  set  of  control  variables  on  the  right-hand  side  of  the  regression.  Wang 
and  San  Miguel  (2012),  a  recent  work  on  defense  contractors’  excessive  profits,  provided  us 
with  a  reference  for  that  purpose. 

Wang  and  San  Miguel  (2012)  not  only  confirmed  the  existence  of  defense 
contractors’  excessive  profits  but  also  they  document  two  determinants  of  excessive 
profitability.  In  particular,  by  showing  defense  contractors’  excessive  profits  being  more 
pronounced  after  1992,  they  argued  that  the  post-1992  significant  industry  consolidation 
improved  the  bargaining  power  of  the  newly  combined  firms  and,  in  turn,  amplified  these 
firms’  profitability.  This  basically  indicates  that  the  degree  of  industry  concentration  is  a  key 
determinant  of  excessive  profit.  The  second  determinant  documented  by  Wang  and  San 
Miguel  (201 2)  is  the  quality  of  corporate  governance,  as  measured  by  the  duality  of  the  chief 
executive  officer  (CEO)  and  the  chairman  of  the  board.  The  main  justification  behind  this 
relation  is  that  poorer  corporate  governance  exacerbates  firms’  rent-seeking  behavior  that 
arises  from  substantial  information  asymmetry  between  the  government  and  defense 
contractors. 

In  addition  to  the  two  determinants  from  Wang  and  San  Miguel  (2012),  that  is,  the 
degree  of  industry  concentration  and  the  quality  of  corporate  governance,  we  also  include 
the  size  of  the  firm  as  a  third  control  variable.  There  are  two  reasons  for  doing  that.  First, 
firm  size  is  a  commonly  used  control  variable  in  empirical  corporate  finance  studies.  The 
justification  is  that  size  is  such  a  “composite”  variable  that  incorporates  so  many 
characteristics  and  information  that  for  any  particular  study,  it  is  a  noisy  measure  of  the 
particular  variable  of  interest,  yet  a  universal  and  perfect  control  variable  that  is  nice  to  be 
included  on  the  right-hand  side.  Second,  Table  3  clearly  shows  that  there  is  a  negative 
correlation  between  the  size  of  the  firm  and  the  firm’s  excessive  profitability,  and  a  positive 
correlation  between  the  size  of  the  firm  and  the  firm’s  political  connection;  that  is,  smaller 
defense  contractors  tend  to  exhibit  more  pronounced  excessive  profits  and  less  political 
connection  relative  to  bigger  ones.  Hence,  it  is  appropriate  to  include  the  size  of  the  firm  as 
a  control  to  avoid  the  potential  correlated  omitted  variable  problem  that  could  damage  the 
statistical  inferences  of  the  multivariate  regression  model. 

So  the  multivariate  regression  includes  three  control  variables  besides  the  variable  of 
interest  (i.e.,  political  connection).  The  dependent  variable  is,  of  course,  the  excessive 
profits  as  defined  by  a  three-way  industry-year-size  matched  excessive  ROA,13  as 
elaborated  in  Wang  and  San  Miguel  (2012).  The  empirical  proxies  for  the  three  control 
variables  are  as  follows:  we  use  a  logarithm  of  total  revenue  as  “firm  size,”  the  duality  of 
CEO  and  chairman  of  the  board  as  a  binary  measure  of  “corporate  governance,”  and  the 


13  Where  industry  is  defined  as  four-digit  SIC  code,  size  is  defined  as  total  assets.  Alternative 
definitions  yield  similar  results. 
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percentage  of  industry  revenue  represented  by  the  largest  four  firms  within  the  industry  as  a 
gauge  of  the  degree  of  industry  concentration.  Same  as  Wang  and  San  Miguel  (2012),  we 
extract  total  revenue  from  Compustat  and  assess  whether  the  CEO  is  also  the  chairman  of 
the  board  from  firms’  proxy  statements.  Regarding  the  proxy  for  the  degree  of  industry 
concentration,  we  use  the  Year-2007  “Concentration  Ratios”  published  by  the  Census 
Bureau  of  the  U.S.  Department  of  Commerce.  Table  4  reports  the  regression  results. 


Table  4.  Multivariate  Regression:  The  Excessive  Profitability  and  Firms’  Political 

Connections 


Dependent  Variable:  Industry-Year-Size  Matched  Excessive  ROA 

Excessive  ROA=  a+  Apolitical  connection  +c*corporate  governance+  d*firm  size+ 

Independent 

e*industry  concentration 

Variables 

Political  Connection  measured  by  a 

Political  Connection  measured  by  the  percent 

dummy  indicator 

of  politically  connected  directors  in  the  Board 

Intercept 

0.05 

0.04 

Political 

-0.04 

-0.07 

Connection 

(t-value) 

(0.01)*** 

(0.04)** 

CEO-Chairman 

0.01 

0.01 

Duality  Dummy 
(t-value) 

(0.29) 

(0.31) 

Firm  Size 

-0.08 

-0.08 

(t-value) 

(0.05)** 

(0.05)** 

Industry 

0.10 

0.11 

Concentration 

(p-value) 

(0.03)** 

(0.02)** 

Notes.  *  indicates  10%  significance  level,  **  indicates  5%  significance  level,  ***  indicates  1% 
significance  level;  CEO-Chairman  dummy  takes  value  of  one  if  the  CEO  is  also  the  chairman;  Firm 
size  is  defined  as  logarithm  of  total  revenue;  Industry  concentration  is  defined  as  the  percentage  of 
industry  revenue  represented  by  the  largest  four  companies  within  the  industry. 


Table  4  shows  that  excessive  profitability  is  lower  for  those  firms  with  political 
connections,  regardless  of  whether  political  connection  is  measured  as  a  binary  indicator 
variable  or  as  the  percentage  of  politically  connected  directors  on  the  board.  The  magnitude 
of  the  impact  is  both  statistically  and  economically  significant.  Moreover,  this  result  holds 
after  controlling  other  known  determinants  of  excessive  profits.  The  signs  of  all  the  control 
variables  are  as  expected,  and  the  magnitudes  of  the  coefficients  of  control  variables  are 
significant  except  for  the  corporate  governance  proxy.  Overall,  the  multivariate  regression 
results  reject  the  corruption  or  rent-seeking  hypothesis  and  suggest  a  non-opportunistic 
motive  of  establishing  political  connections  through  board  directors’  prior  experience. 


Conclusion 

Using  a  slightly  reduced  sample  from  the  one  used  by  Wang  and  San  Miguel  (2012), 
we  investigate  the  impact  of  political  connections  on  excessive  profits  of  defense 
contractors.  We  measure  political  connections  by  searching  the  biographies  of  board 
directors  in  the  firms’  proxy  statements.  We  find  that  defense  contractors  are  more  likely  to 
have  politically  connected  director(s)  in  their  board;  moreover,  among  defense  contractors, 
those  with  a  politically  connected  board  tend  to  have  a  higher  percentage  of  revenue  from 
defense  contracts  than  those  without  political  connection.  While  the  evidence  may  suggest 
that  defense  contractors  have  stronger  incentives  to  establish  political  connections  through 
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the  recruitment  of  board  directors,  and  those  directors  may  indeed  help  the  firm  to  compete 
for  government  contracts,  they  do  not  necessarily  support  a  “rent-seeking”  or  “corruption” 
hypothesis.  In  fact,  in  testing  the  “corruption  hypothesis”  versus  an  alternative  “non- 
opportunistic  motive  hypothesis”  in  the  setting  of  defense  contractors’  excessive  profits,  we 
find  strong  evidence  refuting  the  former  and  in  favor  of  the  latter.  This  suggests  that  defense 
contractors  may  hire  those  politically  connected  directors  and  use  their  experience  to  serve 
a  benevolent  role  to  the  public.  For  instance,  one  legitimate  use  of  the  political  experience  is 
to  keep  defense  contractors  from  opportunistic  profit-seeking  behaviors  that  could  reach  or 
even  cross  federal  government  regulatory  redlines. 
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Abstract 

Dr.  Ashton  Carter,  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD  [AT&L]),  and  Robert  F.  Hale,  Under  Secretary  of  Defense  (Comptroller/Chief  Financial 
Officer),  issued  a  Joint  memorandum  on  April  22,  201 1 ,  titled  Joint  Memorandum  on  Saving 
Related  to  “Should-Cost.”  As  iterated  in  the  memorandum,  Dr.  Carter’s  goal  for  the  should- 
cost  initiative  is  to  ensure  that  program  managers  (PMs)  drive  productivity  improvements  into 
their  programs  during  contract  negotiations  and  throughout  program  execution  and 
sustainment.  This  is  achievable,  according  to  Dr.  Carter,  if  PMs  continuously  perform  should- 
cost  analysis  that  scrutinizes  every  element  of  government  and  contractor  cost. 

In  addition  to  the  Joint  memorandum,  Dr.  Carter  issued  a  second  memorandum  on  April  22, 
2011,  for  acquisition  and  logistics  professionals,  titled  Implementation  of  Will-Cost  and 
Should-Cost  Management.  This  guidance  is  applicable  for  all  acquisition  category  (ACAT)  I, 
II,  and  III  programs. 

The  purpose  of  this  research  is  to  examine  the  potential  impacts  this  and  related  directives 
have  on  the  contracting  community’s  ability  to  request,  acquire,  audit,  and  utilize  data 
germane  to  contract  negotiations  and  management  and  whether  there  may  be  inherent 
potential  conflicts  with  the  commercial  item  acquisition  provisions  of  Federal  Acquisition 
Regulation  (FAR)  Part  12  and  the  contract  pricing  initiatives  of  FAR  Part  15  to  reduce 
reliance  on  the  Truth  in  Negotiations  Act  (TINA)  requirements  for  certified  cost  and  pricing 
data  and  cost  accounting  standards  (CAS),  and  explore  strategies  for  implementing  the 
directive  effectively.  Additionally,  the  research  determines  the  nature  and  extent  of  any 
potential  impacts  on  the  Defense  Contract  Management  Agency  (DCMA)  and  Defense 
Contract  Audit  Agency  (DCAA)  at  supporting  the  should-cost  effort. 

Research  Purpose  and  Objective 

In  response  to  skyrocketing  program,  acquisition,  and  contract  cost  on  major 
weapons  systems,  Dr.  Ashton  Carter,  Under  Secretary  of  Defense  (Acquisition,  Technology, 
and  Logistics;  USD  [AT&L]),  and  Robert  F.  Hale,  Under  Secretary  of  Defense 
(Comptroller/Chief  Financial  Officer),  issued  a  Joint  memorandum  on  April  22,  2011,  titled 
Joint  Memorandum  on  Savings  Related  to  “Should-Cost."  As  iterated  in  the  memorandum, 
Dr.  Carter’s  goal  for  the  should-cost  initiative  is  to  ensure  that  program  managers  (PMs) 
drive  productivity  improvements  into  their  programs  during  contract  negotiations  and 
throughout  program  execution  and  sustainment.  This  is  achievable,  according  to  Dr.  Carter, 
if  PMs  continually  perform  should-cost  analysis  that  scrutinizes  every  element  of 
government  and  contractor  cost. 

In  addition  to  the  Joint  memorandum,  Dr.  Carter  issued  a  second  memorandum  on 
April  22,  2011,  for  acquisition  and  logistics  professionals,  titled  Implementation  of  Will-Cost 
and  Should-Cost  Management.  This  guidance  is  applicable  for  all  acquisition  category 
(ACAT)  1,11,  and  III  programs. 
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The  objective  of  this  research  is  to  examine  the  potential  impacts  this  and  related 
directives  have  on  the  contracting  community’s  ability  to  request,  acquire,  audit,  and  utilize 
data  germane  to  contract  negotiations  and  management  and  to  determine  whether  there 
may  be  inherent  potential  conflicts  with  commercial  item  acquisition  provisions  of  FAR  Part 
12,  and  Contract  Pricing  FAR  Part  15  initiatives  to  reduce  reliance  on  the  Truth  in 
Negotiations  Act  (TINA)  requirements  for  certified  cost  and  pricing  data  and  cost  accounting 
standards  (CAS),  and  explore  strategies  for  implementing  the  directive  effectively. 
Additionally,  the  research  determines  the  nature  and  extent  of  any  potential  impacts  on  the 
Defense  Contract  Management  Agency  (DCMA)  and  Defense  Contract  Audit  Agency 
(DCAA)  at  supporting  the  should-cost  effort  as  iterated. 

It  is  my  belief  that  this  work  will  add  value  to  the  current  body  of  work  designed  to 
create  a  culture  of  efficiency  and  effectiveness  in  Department  of  Defense  (DoD) 
procurement  and  contracting  and  provide  a  highly  referenced  and  readable  work  useful  for 
policy-makers,  practitioners,  and  academics. 

Research  Questions 

The  primary  research  questions  addressed  in  this  paper  are  as  follows: 

•  What  specific  impact  does  Dr.  Carter’s  should-cost  directive  have  on  DoD 
contracting  as  related  to  protocols  for  acquiring  commercial  items? 

•  What  are  the  data  requirement  provisions  under  protocols  for  acquiring 
commercial  items  versus  non-protocols  for  acquiring  commercial  items? 

•  Is  the  should-cost  requirement  approach,  as  defined  in  the  memorandum, 
achievable  under  the  commercial  item  acquisition  provisions  of  the  Federal 
Acquisition  Reform  Act  (FARA)  and  the  Federal  Acquisition  Streamlining  Act 
(FASA),  or  does  the  memorandum  call  for  another  acquisition  strategy  using 
non-protocols  for  acquiring  commercial  items? 

•  If  the  should-cost  memorandum  mandates  are  to  be  achieved,  what  specific 
actions  and  strategies  must  be  taken  by  contracting  offices  to  support  the 
mandate? 

•  Are  the  DCMA  and  DCAA  able  to  fully  support  this  initiative,  and  what  specific 
actions  must  they  take? 

•  What  specific  findings  and  recommendations  can  be  proffered  to  effectively 
implement  the  should-cost  initiatives? 

Methodology  and  Scope 

This  research  includes  a  thorough  literature  review,  examination  and  assimilation  of 
key  policy  documents,  and  outreach  to  subject-matter  experts  (SMEs)  integral  to  the  should- 
cost  will-cost  initiative.  Specific  sources  include,  but  are  not  limited  to,  the  following: 

•  Government  Accountability  Office  (GAO)  reports  and  testimony, 

•  existing  and  ongoing  research  efforts  at  the  Naval  Postgraduate  School 
(NPS), 

•  professional  information  sources  from  major  systems  PM  and  contracting 
activities, 

•  academic  literature,  and 

•  SMEs  within  the  DoD  and  other  organizations. 
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Whenever  SMEs  are  utilized,  the  DoD  and  NPS  mandate  that  Institutional  Review 
Board  (IRB)  protocols  be  followed  to  ensure  SMEs  are  given  full  notification  of  a 
researcher’s  intent  to  use  information  gathered  from  them  for  research  purposes.  In 
accordance  with  these  policies,  I  obtained  consent  from  all  SMEs  that  I  consulted  as  part  of 
my  research  for  this  published  work. 

Based  on  the  information  obtained  through  this  research,  I  make  conclusions  and 
recommendations  to  professionals  desiring  a  better  understanding  of  the  implementation  of 
Dr.  Carter’s  should-cost  will-cost  initiative,  address  concerns  over  potential  conflicts  with  the 
FARA  and  FASA,  and  identify  how  the  DoD  may  be  best  structured  for  achieving  the 
greatest  efficiencies  and  effectiveness  at  implementation. 

Should-Cost  and  Will-Cost  Defined 

The  definitions  of  should-cost  and  will-cost  are  necessary  for  an  understanding  of  the 
concepts  and  their  applicability. 

•  Will-cost  is  defined  as  what  a  program  weapons  system  is  likely  to  cost  given 
a  non-advocate  (independent)  cost  estimate,  such  as  in  an  independent  cost 
estimate  (ICE)  or  independent  government  estimate  (IGE),  based  primarily  on 
historical  cost  incurred. 

•  Should-cost  is  defined  as  the  program  weapons  system  cost  adjusted  for  the 
program’s  initiatives  or  opportunities  to  reduce  cost  below  the  ICE  level. 

The  main  difference  between  will-cost  and  should-cost  is  the  extensive  use  of 
historically  incurred  cost  for  will-cost  estimates  versus  the  examination  of  forward-looking 
efforts  at  reducing  cost  in  operations. 

Better  Buying  Power:  Mandate  for  Restoring  Affordability  and  Productivity  in  Defense 
Spending  (June  2010) 

On  June  28,  2010,  Dr.  Carter  issued  the  first  in  a  series  of  memoranda  mandating 
affordability  and  efficiency  in  DoD  spending.  The  memorandum  for  acquisition 
professionals,  titled  Better  Buying  Power:  Mandate  for  Restoring  Affordability  and 
Productivity  in  Defense  Spending  (Carter,  2010a),  laid  the  foundation  for  all  subsequent 
memoranda  issued  over  the  next  15  months.  In  this  memorandum,  Dr.  Carter  called  for 

delivering  better  value  to  the  taxpayer  and  improving  the  way  the  Department 
does  business.  ...  We  must  abandon  inefficient  practices  accumulated  in  a 
period  of  budget  growth  and  learn  to  manage  defense  dollars  in  a  manner 
that  is,  to  quote  Secretary  Gates  at  his  May  8,  2010  speech  at  the 
Eisenhower  Library,  “respectful  of  the  American  taxpayer  at  a  time  of 
economic  and  fiscal  distress.”  (Carter,  2010a) 

Better  Buying  Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in 
Defense  Spending  and  Implementation  Directive  for  Better  Buying  Power — Restoring 
Affordability  and  Productivity  in  Defense  Spending  (September  2010) 

Dr.  Carter  subsequently  issued  two  memoranda,  again  while  acting  as  USD(AT&L); 
both  memoranda  were  dated  and  released  on  September  14,  2010.  The  first  memorandum 
is  titled  Better  Buying  Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in 
Defense  Spending  (Carter,  2010b)  and  the  second  is  titled  Implementation  Directive  for 
Better  Buying  Power — Restoring  Affordability  and  Productivity  in  Defense  Spending  (Carter, 
2010c). 

The  memorandum  Implementation  Directive  for  Better  Buying  Power — Restoring 
Affordability  and  Productivity  in  Defense  Spending  (Carter,  2010c)  requested  the  Director, 
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Defense  Procurement  and  Acquisition  Policy  (DPAP)  to  develop  the  protocols  and 
manpower  required  to  implement  the  overarching  initiatives  in  the  Better  Buying  Power 
memorandums.  This  request  included  incorporation  and  integration  of  key  agencies  in  the 
protocol  and  manpower  reviews,  including  the  DCMA  and  the  DCAA.  An  excerpt  from  this 
memorandum  states, 

Work  with  the  Defense  Contract  Audit  Agency  (DCAA)  and  the  Defense 
Contract  Management  Agency  (DCMA)  to  develop  guidance,  which  will 
clearly  spell  out  the  roles  and  responsibilities  of  each  organization  in  those 
areas  where  duplication  and  overlap  occur.  Provide  recommended  guidance 
to  me  and  to  the  Under  Secretary  of  Defense  (Comptroller)  by  December  1 , 
2010. 

By  October  1,  2010,  you  are  to  task  DCMA  to  be  responsible  for  the 
promulgation  of  all  Forward  Pricing  Rate  Recommendations.  In  those  cases, 
where  DCAA  has  completed  an  audit  of  a  particular  contractor’s  rates,  DCMA 
shall  adopt  the  DCAA  recommended  rates  as  the  Department’s  position  with 
regards  to  those.  (Carter,  2010c) 

Dr.  Carter  also  stated, 

To  put  it  bluntly:  we  have  a  continuing  responsibility  to  procure  the  critical 
goods  and  services  our  forces  need  in  the  years  ahead,  but  we  will  not  have 
ever-increasing  budgets  to  pay  for  them.  We  must  therefore  strive  to  achieve 
what  economists  call  productivity  growth:  in  simple  terms,  to  DO  MORE 
WITHOUT  MORE.  (Carter,  2010c) 

Acting  on  Secretary  of  Defense  Robert  Gates’  call  for  obtaining  greater  efficiencies  in 
DoD  procurements,  Dr.  Carter  worked  with  senior  leaders  in  the  acquisition  community — 
including  the  component  acquisition  executives  (CAEs),  senior  logisticians  and  systems 
command  leaders,  the  Office  of  the  Secretary  of  Defense  (OSD),  program  executive  officers 
(PEOs),  and  PMs — to  create  the  Better  Buying  Power  initiatives  and  guidance.  The 
guidance  potentially  affected  $400  billion  of  the  $700  billion  DoD  budget  spent  on  goods  and 
services  ($200  billion  each  for  weapons,  electronics,  fuel,  etc.,  and  $200  billion  for 
information  technology  [IT]  support).  Secretary  Gates  and  Dr.  Carter  estimated  the  potential 
savings  from  the  initiatives  and  guidance  as  a  significant  element  of  the  targeted  $100  billion 
from  unproductive  to  more  productive  purposes  over  the  five-year  period  from  201 1-2015. 

Within  the  USD(AT&L)  guidance  memorandum,  the  should-cost  protocol  was 
addressed  as  a  means  to  reduce  unproductive  overhead  within  supporting  contractors  and 
to  capture  reductions  in  contracts  by  informing  future  price  and  contract-type  negotiations 
(Carter,  2010b).  The  following  is  an  excerpt  from  Dr.  Carter’s  September  14,  2010,  Better 
Buying  Power  memorandum: 

During  contract  negotiation  and  program  execution,  our  managers  should  be 
driving  productivity  improvement  in  their  programs.  They  should  be 
scrutinizing  every  element  of  program  cost,  assessing  whether  each  element 
can  be  reduced  relative  to  the  year  before,  challenging  learning  curves, 
dissecting  overheads  and  indirect  costs,  and  targeting  cost  reduction  with 
profit  incentive — in  short,  executing  to  what  the  program  should  cost.  The 
Department’s  decision  makers  and  Congress  use  independent  cost  estimates 
(ICE) — forecasts  of  what  a  program  will  cost  based  upon  reasonable 
extrapolations  from  historical  experience — to  support  budgeting  and 
programming.  While  ICE  Will  Cost  analysis  is  valuable  and  credible,  it  does 
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not  help  the  program  manager  to  drive  leanness  into  the  program.  In  fact,  just 
the  opposite  can  occur:  the  ICE,  reflecting  business-as-usual  management  in 
past  programs,  becomes  a  self-fulfilling  prophecy.  The  forecast  budget  is 
expected,  even  required,  to  be  fully  obligated  and  expended. 

To  interrupt  this  vicious  cycle  and  give  program  managers  and 
contracting  officers  and  their  industry  counterparts  a  tool  to  drive  productivity 
improvement  into  programs,  I  will  require  the  manager  of  each  major  program 
to  conduct  a  Should  Cost  analysis  justifying  each  element  of  program  cost 
and  showing  how  it  is  improving  year  by  year  or  meeting  other  relevant 
benchmarks  for  value.  Meanwhile,  the  Department  will  continue  to  set  the 
program  budget  baseline  (used  also  in  ADMs  and  Selected  Acquisition 
Reports  (SARs))  using  an  ICE.  We  will  use  this  method,  for  example,  to  drive 
cost  down  in  the  Joint  Strike  Fighter  (JSF)  program,  the  Department’s  largest 
program  and  the  backbone  of  tactical  air  power  for  the  U.S.  and  many  other 
countries  in  the  future.  This  aircraft’s  ICE  (Will  Cost)  average  unit  price  grew 
from  $50  million  Average  Unit  Procurement  Cost  (APUC)  when  the  program 
began  (in  2002  dollars,  when  the  program  was  baselined)  to  $92  million  in  the 
most  recent  ICE.  Accordingly,  the  JSF  program  had  a  Nunn-McCurdy  breach 
last  year  and  had  to  be  restructured  by  the  Secretary  of  Defense.  As  a  result 
of  that  restructuring,  a  Should  Cost  analysis  is  being  done  in  association  with 
the  negotiation  of  the  early  lot  production  contracts.  The  Department  is 
scrubbing  costs  with  the  aim  of  identifying  unneeded  cost  and  rewarding  its 
elimination  over  time.  The  result  should  be  a  negotiated  price  substantially 
lower  than  the  Will  Cost  ICE  to  which  the  Department  has  forecasted  and 
budgeted.  Secretary  Gates  indicated  in  his  Efficiency  Initiative  that  the 
Service  that  achieved  the  efficiency  could  retain  monies  saved  in  this  way;  in 
this  case  the  Air  Force,  Navy,  and  Marine  Corps  could  reallocate  JSF  funds 
to  buy  other  capabilities. 

The  Department  will  obligate  about  $2  trillion  in  contracts  over  the  next 
five  years  according  to  Will  Cost  estimates,  so  savings  of  a  few  percent  per 
year  in  execution  are  significant. 

The  metric  of  success  for  Should  Cost  management  leading  to  annual 
productivity  increases  is  annual  savings  of  a  few  percent  from  all  our  ongoing 
contracted  activities  as  they  execute  to  a  lower  figure  than  budgeted.  Industry 
can  succeed  in  this  environment  because  we  will  tie  better  performance  to 
higher  profit,  and  because  affordable  programs  will  not  face  cancellation. 
(Carter,  2010b,  pp.  3-4) 

This  excerpt,  on  close  examination,  promoted  a  forward-looking  analysis  of 
contractors’  embedded  practices  and  associated  cost  for  production  as  the  should-cost 
position  on  which  PMs  must  focus,  rather  than  on  the  initial  and/or  existing  will-cost  position 
that  serves  as  the  initial  baseline  for  the  program. 

Implementation  Directive  for  Better  Buying  Power — Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending  (November  2010) 

Dr.  Carter’s  seven-page  November  3,  2010,  memorandum,  titled  Implementation 
Directive  for  Better  Buying  Power — Obtaining  Greater  Efficiency  and  Productivity  in  Defense 
Spending ,  reiterated  guidance  provided  in  prior  memoranda  and  specified  actions  that  the 
secretaries  of  the  military  departments  and  directors  of  defense  agencies  should  execute 
immediately  or  in  the  time  frame  specified  within  the  memorandum.  The  memorandum  also 
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stated  that  additional  actions  in  support  of  the  initiatives  proffered  in  the  memoranda  dated 
September  14,  2010,  would  be  developed  over  the  following  weeks  and  months.  The 
memorandum  addressed  five  specific  areas  from  the  September  14,  2010,  memoranda:  (1) 
targeting  affordability  and  controlling  cost  growth,  (2)  incentivizing  productivity  and 
innovation  in  industry,  (3)  promoting  real  competition,  (4)  improving  tradecraft  in  service 
acquisition,  and  (5)  reducing  non-productive  processes  and  bureaucracy. 

Will-cost  and  should-cost  are  specifically  addressed  in  the  following  excerpt  from  Dr. 
Carter’s  memorandum: 

Effective  November  15,  2010,  you  will  establish  “Should  Cost”  targets  as 
management  tools  for  all  ACAT  I  programs  as  they  are  considered  for  major 
MS  decisions.  As  described  in  my  September  14,  2010,  Guidance  to  the 
acquisition  workforce,  “Should  Cost”  targets  will  be  developed  using  sound 
estimating  techniques  that  are  based  on  bottom-up  assessments  of  what 
programs  should  cost,  if  reasonable  efficiency  and  productivity  enhancing 
efforts  are  undertaken. 

These  costs  will  be  used  as  a  basis  for  contract  negotiations  and  contract 
incentives  and  to  track  contractor  and  program  executive  officer/project 
manager  performance.  Program  performance  against  “Should  Cost” 
estimates  will  be  reported  to  the  Office  of  Acquisition  Resources  and  Analysis 
through  Acquisition  Visibility  Service  Oriented  Architecture  (AV  SoA). 

By  January  1 ,  201 1 ,  you  will  establish  “Should  Cost”  estimates  for  ACAT 
II  and  III  programs  as  they  are  considered  for  component  MS  decisions.  You 
will  use  “Should  Cost”-based  management  to  track  performance  of  ACAT  II 
and  III  programs.  (Carter,  201  Od) 

Dr.  Carter  further  invoked  the  should-cost  initiative  in  addressing  poor  tradecraft  in 
services  acquisitions,  stating, 

I  will  issue  further  detailed  guidance  for  establishing  taxonomy  of  preferred 
contract  types  in  services  acquisition,  but  starting  immediately,  you  will 
ensure  that  services  acquisitions  under  your  control  are  predisposed  toward 
Cost-Plus-Fixed-Fee  (CPFF)  or  Cost-Plus-Incentive  Fee  (CPIF) 
arrangements  when  robust  competition  or  recent  competitive  pricing  history 
does  not  exist.  This  practice  will  be  used  to  build  sufficient  cost  knowledge  of 
those  services  within  that  market  segment.  You  will  employ  that  cost 
knowledge  to  inform  the  “Should  Cost”  estimates  of  future  price  and  contract 
type  negotiations.  When  robust  competition  already  exists,  or  there  is  recent 
competitive  pricing  history,  you  will  ensure  that  services  acquisitions  under 
your  control  are  predisposed  toward  Firm-Fixed-Price  (FFP)  type  contract 
arrangements.  FFP  should  also  be  used  to  the  maximum  extent  reasonable 
when  ongoing  competition  is  used  in  Multiple  Award  Contract  scenarios. 
(Carter,  2010c) 

In  the  preceding  context,  Dr.  Carter  wanted  to  build  a  knowledge  base  of  cost  within 
particular  service  segments  where  true  competition  is  not  driving  the  prices  paid.  This  can 
only  be  accomplished  through  contract  vehicles  that  allow  for  detailed  submission  of  cost 
estimates  in  discussions  and  negotiations  and  for  utilization  of  that  data  to  support  future 
contract  negotiations.  Hence  Dr.  Carter’s  predisposition  for  cost-plus-fixed-fee  (CPFF)  and 
cost-plus-incentive-fee  (CPIF)  contract  arrangements  in  non-competitive  circumstances. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-446- 


Programs  Initially  Covered  by  Ashton  Carter’s  Should-Cost  Initiative 

The  implementation  of  will-cost  and  should-cost  management  initiatives  was  targeted 
at  five  ACAT  l-lll  programs  equally  allocated  in  the  Army,  Navy,  and  Air  Force.  The  five 
programs  (shown  in  Table  1 )  vary  in  their  current  maturity  and  milestone  attainments. 


Table  1.  Should-Cost  Management  Example  (Pilot)  Programs 


Air  Force 

Army 

Navy 

Joint  Strike  Fighter  (F-35) 

Joint  Air  Ground  Missile 
(JAGM) 

Joint  Strike  Fighter  (F-35) 

Global  Hawk  Blocks  30  & 

40  (GH  BLK  30  &  40) 

Black  Hawk  (UH-60M) 

Hawkeye  (E-2D) 

Space  Based  Infrared 
System  (SB IRS) 

Ground  Combat  Vehicle 
(GCV) 

Presidential  Helo  (VXX) 

Evolved  Expendable 
Launch  Vehicle  (EELV) 

Paladin  Product 
Improvement  (PIM) 

Littoral  Combat  Ship 
(LCS) 

Advanced  Extremely 

High  Frequency  (AEHF) 
Satellite  System 

NETT  Warrior 

Ohio  Replacement 
Program 

Note.  The  information  in  this  table  was  adapted  from  Dr.  Carter’s  Implementation  of  Will-Cost  and 
Should  Cost  Management  memorandum,  dated  April  22,  201 1 . 


Should-Cost  Will-Cost  Implementation  Memoranda  Summary 

The  Services,  Navy,  Air  Force,  and  Army,  have  implemented  Dr.  Carter’s  should-cost 
initiative  with  striking  similarities.  Table  2  is  an  examination  of  the  implementation 
memoranda  key  elements  and  provisions. 


Table  2.  Implementation  Memoranda  Key  Elements  and  Provisions 

(Yoder,  2012) 


Key  Common 

Element 

Navy 

Implementation — 

ASN  (RD&A)  Memo 
July  19,  2011 

Air  Force 
Implementation — 
Dept,  of  the  Air 

Force  Memo  June 

15, 2011 

Army 

Implementation — 
Dept,  of  Army 

Memo  June  10, 

2011 

Identification  of 
Programs 

Yes 

Yes 

Yes 

Definition  &  Use  of 
Will-Cost 

Yes.  Independent 
baseline  for  program 
budget  and  funding. 
External 
promulgation 
allowed. 

Yes.  Independent 
baseline  for 
program  budget  and 
funding.  External 
promulgation 
allowed. 

Yes.  Independent 
baseline  for 
program  budget  and 
funding.  External 
promulgation 
allowed. 

Development  of 
Will-Cost  Protocols 

Yes.  CAPE  ICE  or 
service  cost  position. 
SECNAVINST 

5223.3  DON  SCP 
germane.  Will-cost  is 
the  program  of 
record  estimate  and 
the  cost  analysis 
requirements 

Yes.  Non-advocate 
baseline  developed 
with  Air  Force  AFPD 
65-5  and  AFI  65- 
508  for  ACAT  1  and 
with  approval  from 
product  or  logistics 
center  financial  cost 
estimating 

Yes.  ICE  existing 
ACAT  1  and 
managed  ACAT  II 
defined  protocols 
extend  to  ACAT  III 
programs.  Will-cost 
estimates  used  for 
baselines  for 
budgeting, 
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description  (CARD). 

organization  (FMC). 

programming,  and 
reporting. 

Definition  &  Use  of 
Should-Cost 

Yes.  PM  develops 
targets  using 
technical  and 
schedule  baselines 
with  applied 
efficiencies,  lessons 
learned,  and  best 
practices  in 
productivity  and  for 
informed 

negotiations  under 
FAR  15.407-4  and 
DFARS  215.407-4. 
External 

promulgation  NOT 
allowed. 

Yes.  PM  develops 
targets  via  driving 
leanness  at  major 
milestone  decisions. 
NOT  used  for 
budgeting, 
programming,  or 
reporting  outside 
the  department. 

Yes.  PM  drives 
leanness  through 
should-cost 
management. 
Incentivizes  targets 
to  performance. 

NOT  for  budgeting, 
programming,  or 
reporting  outside 
the  department. 
Creates  informed 
negotiations  under 
FAR  15.407-4  and 
DFARS  215.407-4. 

Development  of 
Should-Cost 

Targets 

PM  responsible  for 
targets. 

Developed  in  one  or 
more  of  three  ways: 

1)  will-cost  base  with 
discrete, 
measureable 
savings. 

Recommended  for 
all  programs  with  a 
will-cost  estimate. 

2)  bottom-up 
estimate  without  a 
formal  FAR/DFARS 
should-cost  review. 

3)  bottom-up 
estimate  with  a 
formal  FAR/DFARS 
should-cost  review. 

PM  responsible  for 
targets  along  with 
tracking  and 
reporting.  AT&L 
(ACAT  1 D  and 
lAMs)  and  SAF/AQ 
(or  delegated 
PEO/DAO)  approve 
should-cost 
estimates  at 
milestones. 

PM  responsible  for 
identifying  savings 
opportunities  and 
targets.  Not 
applicable  to  quick 
reaction  capabilities. 
PM  determines 
discrete  and 
measurable  targets 
while  maintaining 
realistic  technical 
requirements  and 
schedule.  MDA 
approves  should- 
cost  targets. 
Recommended 
approaches: 

(1)  will-cost  base 
applying  discrete 
measurable 
items/initiatives. 

(2)  bottom-up 
approach  without  a 
detailed 

FAR/DFARS 
should-cost  review. 

(3)  bottom-up  with  a 
formal  FAR/DFARS 
should-cost  review. 

Participants  in 
Should-Cost  Target 

SYSCOM/PM.  May 
seek  assistance 

PM  with  cross¬ 
functional  teams. 

PM  with  assistance 
from  outside 
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Development 

from  the  Naval 

Center  for  Cost 
Analysis  (NCAA), 
DCMA,  and  other 

PM  offices. 

Can  seek 
assistance  from 
outside:  the  AFCAA 
or  DCMA. 

organizations  such 
as  Deputy  Assistant 
Secretary  for  the 
Army  Cost  and 
Economics  (DASA 
[CE])  and  DCMA. 

Milestone  A 

Will-cost  estimate 
(initial  or  updated) 
should-cost 
management  target 
(initial  or  update) 

Will-cost  estimate 
(initial  or  updated) 
should-cost 
management  target 
(initial  or  update) 

Will-cost  estimate 
(initial  or  updated) 
should-cost 
management  target 
(initial  or  update) 

Milestone  B 

Will-cost  update 
(initial  baseline  for 
Nunn-McCurdy 
metrics) 

Should-cost  (sets 
internal  program 
execution  baseline) 
Initial  to  support 
contract  actions 
(optional) 

Will-cost  update 
(initial  baseline  for 
Nunn-McCurdy 
metrics) 

Should-cost  (sets 
internal  program 
execution  baseline) 
Initial  to  support 
contract  actions 
(optional) 

Will-cost  update 
(initial  baseline  for 
Nunn-McCurdy 
metrics) 

Should-cost  (sets 
internal  program 
execution  baseline) 
Initial  to  support 
contract  actions 
(optional) 

Milestone  C 

Update  will-cost  and 
should-cost. 
Indirect/direct 
contract  cost  reviews 
(optional) 

FAR  15.407-4  and 
DFARS  215.407-4 

Update  will-cost  and 
should-cost. 
Indirect/direct 
contract  cost 
reviews  (optional) 
FAR  15.407-4  and 
DFARS  215.407-4 

Update  will-cost  and 
should-cost. 
Indirect/direct 
contract  cost 
reviews  (optional) 
FAR  15.407-4  and 
DFARS  215.407-4 

Full-Rate 

Production 

Decision/Contract 

Update 

Update 

Update 

Withholding  and 
Distribution  of 

Funds 

Yes,  delta  withheld. 
SAEfor  ACATI, 

MDA  for  ACAT  II, 
PEOforACAT  III 

Yes,  delta  withheld. 
Remains  in  program 
element.  Release 
by 

service/component 
acquisition 
executive  (S/CAE) 

Yes,  delta  managed 
consistent  with  the 
type  of  contracts 
used  in  the 
program.  When 
fixed-price  contracts 
are  utilized,  any 
delta  should  be 
considered 
“realized”  and  built 
into  the  contract. 

Reporting 

Templates 

Yes 

Yes 

Yes 

Analysis  of  the  Potential  Impacts  of  Ashton  Carter’s  “Should-Cost”  Memorandum  on 
Defense  Contracting — Findings  and  Recommendations 

The  following  summarizes  key  findings  and  recommendations  presented  in  NPS-CM 
12-199  (Yoder,  2012): 
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•  Finding  &  Recommendation  #1:  FARA  and  FASA 

There  is  a  conflict  in  the  specific  definition  of  commercial  item  acquisition  that 
allows  for  major  weapons  systems  procurements  in  limited-  or  non¬ 
competitive  marketplaces  to  be  characterized  as  commercial  under  FARA 
and  FASA  statutes.  Current  legislative  proposals  are  under  congressional 
review  to  revise  the  statutory  definition. 

•  Finding  &  Recommendation  #2:  Personnel 

The  DCMA,  the  DCAA,  and  the  Services  have  made,  and  are  re-capitalizing 
their  workforce  with  credentialed  personnel  in  key  functional  specialties 
needed  to  support  the  should-cost  initiative.  Key  functional  specialties 
include,  but  are  not  limited  to,  auditors  and  production  specialists,  with 
additional  specialties  in  Lean  Six  Sigma,  process  management,  and  so  forth. 
The  personnel  increases  must  be  protected  against  any  potential  cuts  to 
ensure  that  cost  consciousness  and  reduction  in  systems  acquisition  cost  can 
mature  and  flourish — continue  to  re-capitalize  the  workforce. 

•  Finding  &  Recommendation  #3:  Platforms 

The  CBAR  data  system  has  recently  been  deployed  by  DCMA.  This  platform 
was  established  in  March  201 1 ,  providing  necessary  single-point  access  to 
key  information  spanning  DoD-wide  contracts  and  relevant  information 
required  for  contracting  officers  to  produce  pre-negotiation  business 
clearances,  sometimes  known  as  business  clearance  memoranda  (BCM),  as 
a  pre-cursor  to  conducting  negotiations  pursuant  to  a  contract  award,  and 
data  for  the  continued  management  of  contracts  with  real-time  actionable 
information  available  24/7  via  a  secure  network.  Although  the  DCMA  and 
DCAA  will  drive  much  of  the  data  input,  all  DoD  services  and  systems 
commands  will  have  it,  and  have  key  roles  in  populating  and  managing  data 
in  the  system.  The  CBAR  system  must  be  funded  to  maintain  accurate  and 
recent  data.  The  data  must  be  relevant  and  germane  to  the  should-cost 
effort,  which  will  take  quality  personnel  to  define,  collect,  and  populate  the 
data.  Continued  management  and  maintenance  of  this  system  is  imperative 
and  must  have  high-level  support. 

•  Finding  #4:  Protocols 

Notwithstanding  the  FARA  and  FASA  findings  and  recommendations 
mentioned  previously,  the  protocols  for  should-cost  analysis  have  been 
promulgated  with  an  emphasis  on  flexibility.  This  flexibility  allows  program 
offices  the  highest  degree  of  latitude  in  determining  should-cost  targets  and 
how  to  achieve  those  targets.  That  information  must  be  shared  within  the 
government  for  future  target  savings  and  contract  negotiations.  Continue  to 
emphasize  Service  program  office  entrepreneurship  at  developing  individual 
targets.  Share  information,  internally,  with  other  program  and  contracting 
offices  via  the  CBAR. 

•  Finding  &  Recommendation  #5:  Should-Cost  Target  Savings  Holdback 

There  is  concern  that  if  not  managed  properly,  holdback  funds  may  be  re¬ 
allocated  for  purposes  other  than  improvements  in  immediate  weapons 
systems  acquisition,  thus  creating  a  huge  disincentive  for  program  offices  to 
set  aggressive  should-cost  targets.  Senior  leaders  must  provide  incentives  for 
the  program  offices  to  set  aggressive  should-cost  targets,  wherein  the  will- 
cost  versus  should-cost  potential  savings  have  a  guaranteed  amount  or 
percentage;  I’ll  call  it  a  cost  savings  incentive  (CSI)  that  can  be  used  for 
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program  purposes  and  objectives.  The  program  office  can  utilize  the  CSI 
amount,  which  perhaps  represents  either  the  entire  delta  or  a  portion  of  it. 

•  Finding  &  Recommendation  #6:  Metrics  and  Determining  Success 

Meaningful  metrics  to  determine  the  efficacy  of  the  should-cost  initiative  are 
needed  by  Milestone  authorities,  PMs  and  PCOs,  although  these  metrics 
have  yet  to  be  developed  and  universally  promulgated.  Sound  metrics  for 
cost  reductions,  efficiency  gains  and  such,  must  be  developed  and 
implemented  to  determine  the  efficacy  of  the  should-cost  initiative.  At  a 
minimum,  an  ROI  can  be  developed  and  utilized,  capturing  the  DoD’s  total 
loaded  labor  cost  to  conduct  the  should-cost  efforts,  including  organic  and 
contractor  personnel  dedicated  to  the  efforts,  against  actual  target  savings 
achieved. 

Final  Thoughts  and  Further  Reading 

An  Analysis  of  the  Potential  Impacts  of  Ashton  Carter’s  “Should-Cost”  Memorandum 
on  Defense  Contracting  (NPS-CM-12-199;  Yoder,  2012),  dated  September  17,  2012,  is 
much  more  comprehensive  in  its  presentation  of  this  topic.  The  original  work,  NPS-CM-12- 
199,  contains  77  pages  of  presentation  and  analysis,  along  with  an  additional  95  pages  of 
supporting  appendices,  for  a  total  of  nearly  175  pages — far  more  detailed  than  the 
information  it  is  possible  to  present  in  this  synoptic  examination. 

Those  interested  in  this  topic,  and  those  who  would  like  additional  details,  are 
encouraged  to  access  NPS-CM-12-199  at  the  Naval  Postgraduate  School,  Acquisition 
Research  Program  website  (www.acquisitionresearch.org). 
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Abstract 

Extensive  cost  overruns  in  major  defense  programs  are  common,  and  studies  have  identified 
poor  cost  estimation  as  a  main  contributor.  Research  and  experience  have  identified  several 
factors  associated  with  poor  cost  estimates.  These  include 

•  optimistic  expectations  about  the  program  scope  and  technology  that  can  be 
delivered  on  schedule  and  within  budget; 

•  the  enormous  amount  of  unknowns  and  uncertainty  that  exist  when  these 
estimates  are  made  about  large-scale,  unprecedented  systems  that  take  years  to 
develop  and  deploy;  and 

•  the  heavy  reliance,  of  necessity,  on  expert  judgment. 

In  this  paper,  we  describe  a  new,  integrative  approach  for  pre-Milestone  A  cost  estimation 
called  quantifying  uncertainty  in  early  life  cycle  cost  estimation  (QUELCE).  QUELCE 
synthesizes  scenario  building,  Bayesian  belief  network  (BBN)  modeling,  and  Monte  Carlo 
simulation  into  an  estimation  method  that  quantifies  uncertainties,  allows  subjective  inputs, 
visually  depicts  influential  relationships  among  change  drivers  and  outputs,  and  assists  with 
explicit  description  and  documentation  underlying  an  estimate.  We  use  scenario  analysis  and 
dependency  structure  matrix  (DSM)  techniques  to  limit  the  combinatorial  effects  of  multiple 
interacting  program  change  drivers  to  make  modeling  and  analysis  more  tractable. 

Finally,  we  describe  results  and  insights  gained  from  applying  the  method  retrospectively  to  a 
major  defense  program. 

Background 

The  inaccuracy  of  cost  estimates  for  developing  major  Department  of  Defense  (DoD) 
systems  is  well  documented,  and  cost  overruns  have  been  a  common  problem  that 
continues  to  worsen  (GAO,  2011,  2012).  Because  estimates  are  now  prepared  much  earlier 
in  the  acquisition  life  cycle,  well  before  concrete  technical  information  is  available,  they  are 
subject  to  greater  uncertainty  than  they  have  been  in  the  past  (RAND,  2007).  Early  life 
cycle  cost  estimates  are  often  based  on  a  desired  capability  rather  than  a  concrete  solution. 
Faced  with  investment  decisions  based  primarily  on  capability,  several  problems  emerge 
when  creating  estimates  at  this  early  stage  (Roper,  2010): 

•  Limited  Input  Data:  The  required  system  performance,  the  desired 
architecture  of  the  solution,  and  the  capability  of  the  vendors  are  not  fully 
understood. 

•  Uncertainties  in  Analogy-Based  Estimates:  Most  early  estimates  are  based 
on  analogies  to  existing  products.  While  many  factors  may  be  similar,  the 
execution  of  the  program  and  the  technology  used  as  part  of  the  system  or  to 
develop  it  are  often  different.  For  example,  software  product  size  depends 
heavily  on  the  implementation  technology,  and  the  technology  heavily 
influences  development  productivity.  Size  and  productivity  are  key 
parameters  for  cost  estimation. 

•  Challenges  in  Expert  Judgment:  Wide  variation  in  judgment  can  exist 
between  experts,  and  the  confidence  in  the  input  that  they  provide  is 
generally  not  quantified  and  unknown. 
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•  Unknown  Technology  Readiness:  Technology  readiness  may  not  be  well 
understood,  and  is  likely  to  be  over-  or  underestimated. 

This  paper  describes  the  QUELCE  method  and  experiences  to  date. 

An  Improved  Method  for  Early  Life  Cycle  Cost  Estimation 

The  quantifying  uncertainty  in  early  life  cycle  cost  estimation  (QUELCE)  method  is  an 
integrative  approach  for  pre-Milestone  A  cost  estimation  to  address  the  problems 
associated  with  early  life  cycle  cost  estimation  while  at  the  same  time  providing  benefits  not 
found  in  current  cost  estimation  methods  (Ferguson  et  al. ,  2011).  The  method  aims  to 
provide  credible  program  cost  estimates  as  distributions  rather  than  point  estimates. 
QUELCE  produces  intuitive  visual  representations  of  the  data  that  explicitly  model  influential 
relationships  and  interdependencies  among  the  drivers  on  which  the  estimates  depend. 
Assumptions  and  constraints  underlying  the  estimates  are  well  documented,  which 
contributes  to  better  management  of  cost,  schedule,  and  adjustments  to  program  scope  as 
more  is  learned  and  conditions  change.  Documenting  the  basis  of  an  estimate  facilitates 
updating  the  estimate  during  program  execution  and  helps  others  make  informed  judgments 
about  estimation  accuracy. 

The  QUELCE  method  differs  from  existing  methods  because  it 

•  uses  available  information  not  normally  employed  for  program  cost 
estimation, 

•  explicitly  models  uncertainty  on  the  input  side  of  the  cost  estimation  equation 
in  terms  of  program  change  drivers, 

•  enables  calculation  (and  re-calculation)  of  the  cost  impacts  caused  by 
changes  that  may  occur  during  the  program  life  cycle,  and 

•  enhances  decision-making  through  the  transparency  of  the  assumptions 
going  into  the  cost  estimate. 

Figure  1  shows  the  flow  of  information  in  a  typical  major  defense  acquisition  program 
(MDAP)  acquisition,  with  blue  boxes  added  to  represent  the  contributions  from  the  QUELCE 
method. 
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Figure  1.  Information  Flow  for  Early  Life  Cycle  Estimation,  With  QUELCE 

Method  Additions 


The  QUELCE  Method 

QUELCE  synthesizes  scenario  building,  Bayesian  belief  network  (BBN)  modeling, 
and  Monte  Carlo  simulation  into  an  estimation  method  that  quantifies  uncertainties,  allows 
subjective  inputs,  visually  depicts  influential  relationships  among  change  drivers  and 
outputs,  and  assists  with  the  explicit  description  and  documentation  underlying  an  estimate. 

It  uses  scenario  analysis  and  dependency  structure  matrix  (DSM;  Lindemann,  n.d.) 
techniques  to  eliminate  cycling  among  the  interacting  program  change  drivers  to  make 
modeling  and  analysis  more  tractable.  Representing  scenarios  as  BBNs  enables  sensitivity 
analysis,  exploration  of  alternatives,  and  quantification  of  uncertainty. 

The  BBNs  and  Monte  Carlo  simulation  are  used  to  predict  variability  of  what  become 
the  inputs  to  the  existing  cost  estimation  models  and  tools.  As  a  result,  interim  and  final  cost 
estimates  are  represented  as  distributions  so  that  the  decision-maker  can  see  the  probability 
of  a  program  exceeding  the  specified  cost.  The  method  can  be  described  as  a  series  of  five 
activities,  summarized  in  the  following  sections.2 

Identify  Program  Change  Drivers 

The  identification  of  program  change  drivers  is  best  accomplished  by  the  experts 
who  provide  programs  with  information  about  acquisition,  development,  and  the  technical 
approach,  in  addition  to  direct  input  for  cost  estimation.  A  workshop  setting  is  used  to 
identify  drivers  that  could  affect  program  costs.  These  experts  consider  all  aspects  of  a 
program  that  might  change  and  significantly  affect  its  execution  during  the  program’s  life 
cycle — particularly  given  the  new  information  developed  during  the  Technology 
Development  Phase  in  preparation  for  Milestone  B.  The  probability  of  program  success 
(POPS)  factors  used  by  the  Navy  and  Air  Force  can  be  used  to  start  the  brainstorming  and 
discussion. 


2  This  work  was  originally  described  in  a  two-part  series  on  the  SEI  blog,  A  New  Approach  for 
Developing  Cost  Estimates  in  Software  Reliant  Systems  (http://blog.sei.cmu.edu/post.cfm/improving- 
the-accuracy-of-early-cost-estimates-for-software-reliant-systems-first-in-a-two-part-series). 
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In  support  of  this  step,  we  have  found  that  there  is  much  useful  information  contained 
in  a  variety  of  documents  produced  during  the  pre-Milestone  A  phase.  These  include  the 
Analysis  of  Alternatives  and  the  various  reports  and  documents  developed  as  part  of  the 
Materiel  Solution,  the  Technology  Development  Strategy,  and,  where  available,  any  pre- 
Milestone  A  assessments  such  as  the  POPS  gate  reviews.  While  these  traditionally  have 
not  been  considered  for  cost  estimation  purposes,  during  the  conduct  of  a  retrospective 
study,  we  found  these  and  other  program  documents  to  contain  relevant  information 
suggesting  several  program  change  factors.  Our  initial  list  totaled  nearly  60  factors. 

In  the  workshops,  experts  are  asked  to  provide  judgments  about  the  status  of  each 
program  change  driver.  The  specific,  assumed  state  as  proposed  by  the  Materiel  Solution 
and  Technology  Development  Strategy  is  identified  and  labeled  as  the  nominal  state. 

Experts  then  brainstorm  about  possible  changes  in  the  condition  of  each  driver  that  may 
occur  during  the  program  life  cycle.  The  experts  identify  possible  changes  that  might  occur 
to  the  nominal  state  and  use  their  best  judgment  for  the  probability  that  the  nominal  state  will 
change. 

Identify  Interdependencies  and  Reduce  Complexity 

Once  the  changed  conditions — referred  to  as  potential  driver  states — are  fully 
identified,  participants  subjectively  evaluate  the  cause  and  effect  relationships  among  the 
drivers.  Expert  judgment  is  applied  to  rank  the  causal  effects.  A  matrix  is  developed  that 
provides  the  relationship  between  nominal  and  dependent  states  and  contains  the 
conditional  probability  that  one  will  affect  the  other,  but  not  the  impact  of  the  change.  This 
exercise  can  result  in  a  very  large  number  of  program  change  drivers  and  states  identified 
for  an  MDAP. 

Using  dependency  structure  matrix  (DSM;  Lindemann,  n.d.)  techniques,  the  highly 
rated  change  drivers  in  the  matrix  can  be  reduced  to  an  efficient  set  that  has  the  most 
potential  impact  to  program  execution  and,  hence,  cost.  The  DSM  technique  is  a  well- 
established  method  to  reduce  complicated  dependency  structures  to  a  manageable  size. 
Furthermore,  the  technique  helps  to  eliminate  cycles  in  the  matrix  by  transforming  the  matrix 
to  an  upper-right  triangle  and  makes  it  directly  useful  for  constructing  the  BBN.  An  example 
of  a  dependency  matrix  after  DSM  transformation  created  during  an  SEI  workshop  is 
provided  in  Figure  2. 
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Figure  2.  Example  Dependency  Matrix  After  DSM  Transofrmation 


Construct  a  Bayesian  Belief  Network 

A  BBN  is  constructed  using  the  program  change  drivers  derived  from  the  expert 
workshop  and  their  cause-and-effect  relationships.  The  BBN  models  the  change  drivers  as 
nodes  in  a  quantitative  network  and  includes  the  conditional  probabilities  that  changes  of 
state  in  one  node  will  create  a  change  of  state  in  another  node,  as  envisioned  by  the 
program  domain  experts.  Figure  3  depicts  an  abbreviated  visualization  of  a  BBN,  with 
circled  nodes  representing  program  change  drivers  and  arrows  representing  either  cause- 
and-effect  relationships  or  leading  indicator  relationships.  This  example  shows  that  a 
change  in  the  Mission  &  CONOPS  driver  will  likely  cause  a  change  to  the  Capability 
Analysis  driver,  which  in  turn  will  likely  change  the  Key  Performance  Parameters  (KPPs) 
driver  and  subsequently  the  Technical  Challenge  outcome  factor.  The  three  outcome  factors 
(Product  Challenge,  Project  Challenge,  and  Size  Growth)  and  their  corresponding  states  are 
mapped  to  some  of  the  traditional  cost  model  input  factors  and  their  values. 
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Conditional  probabilities  are  assigned  to  the  nodes  (drivers)  in  the  BBN.  Each  node 
can  assume  a  variety  of  states  with  an  associated  likelihood  identified  by  the  domain 
experts.  This  allows  the  calculation  of  outcome  distributions  on  the  variables. 


Domain  experts  use  the  BBN  to  define  scenarios.  The  realization  of  a  potential  state 
in  a  particular  node  is  specified,  and  the  cascading  impacts  to  other  nodes  and  the  resulting 
change  in  the  outcome  variables  are  recalculated.  Any  change  in  one  or  more  nodes 
(drivers)  constitutes  a  scenario.  Once  the  experts  are  satisfied  that  a  sufficient  number  of 
scenarios  are  specified,  they  use  their  judgment  to  rank  them  for  likely  impacts  to  cost.  An 
example  scenario  created  during  an  SEI  pilot  workshop  is  provided  in  Figure  4. 
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Figure  4.  A  Partial  Example  of  a  Scenario  With  Two  Driver  Nodes  in  a  Nominal 

State 


Select  Cost  Estimating  Models  to  Generate  an  Estimate 

Parametric  cost  estimation  models  for  software  use  a  mathematical  equation  to 
calculate  effort  and  schedule  from  estimates  of  size  and  a  number  of  parameters.  A 
decision  is  made  as  to  which  cost  estimating  tools,  cost  estimating  relationships  (CERs),  or 
other  methods  will  be  used  to  form  the  cost  estimate.  COCOMO  II  is  a  well-known 
estimation  tool  and  is  open  source.  The  SEI  has  so  far  developed  the  relationships  between 
BBN-modeled  program  change  drivers  and  COCOMO,  shown  in  Figure  5.  The  red  X’s  in 
brackets  indicate  an  inverse  relationship  between  the  BBN  output  factor  and  the 
corresponding  COCOMO  II  driver.  The  black  X’s  indicate  a  positive  relationship.  The  BBN 
interface  to  the  commercial  SEER-SEM  cost  estimating  tool  is  currently  underway. 
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Figure  5.  Mapping  BBN  Outputs  to  COCOMO  Inputs 


The  program  office  estimates  of  size  and  other  cost  model  inputs  such  as 
productivity  are  used  as  the  starting  point  in  this  step.  Often  these  values  are  estimated  by 
analogy  and  aggregation.  They  are  adjusted  by  applying  the  distributions  calculated  by  the 
BBN. 

Monte  Carlo  Simulation 

From  each  selected  scenario,  we  use  the  output  of  the  BBN  to  parameterize  a  Monte 
Carlo  simulation  of  the  inputs  to  the  selected  cost  estimation  model.  This  provides 
probability  distributions  for  the  input  factors  to  the  cost  estimating  models.  This  also  provides 
explicit  confidence  levels  for  the  results.  Figure  6  shows  the  simulation  results  that  the  SEI 
obtained  when  modeling  a  factor  (person-months)  in  three  different  scenarios. 
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Figure  6.  Simulation  Results  for  Three  Scenarios 

A  report  with  the  final  cost  estimates  is  generated  for  each  scenario,  including  the 
nominal  (expected)  program  plan.  The  explicit  confidence  levels  and  the  visibility  of  all 
considered  program  change  drivers  allow  for  quick  comparisons  and  future  re-calculations. 
This  method  enables  the  creation  of  comparative  scenario  calculations  at  any  point  during 
the  life  cycle.  The  visibility  of  the  program  change  drivers  and  the  transparency  afforded  by 
the  consideration  of  alternative  scenarios — and  their  assumptions — enables  improved 
decision-making  and  contingency  planning. 


Results  and  Future  Research 

To  date,  there  have  been  two  empirical  thrusts  to  the  research.  First,  we  have 
conducted  a  retrospective  on  an  MDAP.  We  constructed  a  10-year  time  line  of  the  program 
using  archival  documents,  records  from  various  DoD  repositories,  and  collaborations  with 
SEI  staff  who  worked  on  the  program. 


The  team  accessed  over  4,100  program  files,  which  documented  virtually  all  of  the 
program’s  history.  In  addition,  the  team  obtained  over  100  official  contractor  submissions  of 
Software  Resource  Data  Reports  (SRDRs)  and  Earned  Value  Management  Reports 
contained  in  the  Defense  Automated  Cost  Information  System.  We  also  obtained  acquisition 
reports  from  the  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  Purview 
repository,  which  included  the  relevant  Selected  Acquisition  Reports  (SARs)  and  the 
Defense  Acquisition  Executive  Summary  (DAES)  Reports. 
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With  the  participation  of  two  in-house  experts  who  had  worked  with  the  program,  we 
established  a  provisional  set  of  57  program-specific  change  drivers.  We  also  elicited  their 
judgment  on  the  likelihood  of  change  for  each  of  the  program  change  drivers  and  their 
potential  cascading  effect  on  the  other  drivers.  These  judgments  formed  the  basis  for 
implementing  DSM  techniques  to  reduce  the  complexity  and  capture  the  cascading  effects 
of  the  interdependencies  among  the  program  change  drivers. 

DSM  reduced  the  number  of  program  change  drivers  to  those  that  the  experts 
considered  to  have  moderate  or  high  likelihood  of  change  during  program  execution.  While 
the  matrix  manipulation  techniques  will  often  remove  many  of  the  cycles  in  the  matrix,  expert 
judgment  is  also  required  to  eliminate  cycles  that  are  not  removed  by  the  algorithms  and 
rating  criteria.  In  the  context  of  this  retrospective,  we  also  realized  that  asking  the  experts  to 
mentally  reconstruct  what  potential  changes  might  have  been  considered  at  the  early  stages 
of  the  program  did  not  avoid  problems  in  bias  based  on  later  experience.  But  if  implemented 
at  pre-Milestone  A  as  envisioned,  these  judgments  represent  the  reality  of  the  early  life 
cycle  estimation  process.  In  the  end,  we  were  left  with  30  program  change  drivers  that 
formed  the  acyclic  graph  required  for  the  construction  of  the  BBN. 

In  assigning  the  required  conditional  probabilities  for  the  BBN  to  each  change  driver, 
we  utilized  both  the  experts’  elicited  judgments  of  probability  and  the  ranges  of  variance 
produced  from  the  expert  calibration  experiments  performed  earlier.  The  elicited 
probabilities  were  used  to  directly  populate  some  portions  of  the  BBN.  However,  we  quickly 
realized  that  it  was  not  feasible  to  elicit  all  of  the  probabilities  and  conditional  probabilities 
required  for  such  a  complex  BBN.  Hence,  we  adapted  an  algorithmic  approach  to 
specifying  the  needed  probabilities.  To  represent  the  uncertainty  in  the  elicited  probabilities 
and  to  incorporate  this  into  the  computed  probabilities,  we  used  the  second  element  noted 
earlier,  the  ranges  of  variance  produced  by  experiments  conducted  to  calibrate  expert 
judgment  to  a  90%  confidence  range.  This  calibration  research  is  the  second  thrust  of  this 
work  and  is  documented  in  a  separate  technical  report  (Goldenson  &  Stoddard,  2013). 

For  purposes  of  demonstration,  we  relied  on  using  the  results  of  those  experiments. 
However,  in  a  “live  action”  MDAP,  we  would  use  the  actual  program  experts’  calibration 
results,  which  would  be  obtained  through  a  calibration  test.  The  technical  workshop  with  the 
MDAP  experts  would  then  serve  to  both  elicit  their  required  judgments  as  described  earlier 
and  allow  them  to  participate  in  a  series  of  calibration  training  exercises.  The  exercises 
sharpen  expert  abilities  to  exert  less  overconfident  and  less  overoptimistic  judgment  while 
also  producing  the  required  data  for  us  to  capture  uncertainty  within  the  BBN. 

The  resulting  retrospective  BBN  enabled  the  output  of  probability  distributions  used 
as  inputs  to  the  cost  estimation  tool.  We  constructed  linkages  to  the  SEER-SEM  cost 
estimation  tool  used  by  the  program  for  the  system  software  components  comprising  it. 
Monte  Carlo  techniques  allowed  us  to  generate  confidence  intervals  for  these  distributions, 
which  were  then  used  for  input  to  the  cost  model. 

We  are  close  to  completing  the  retrospective  and  will  be  comparing  the  results  of  the 
QUELCE  model  with  the  estimates  and  actual  costs  produced  by  the  program.  The  conduct 
of  the  retrospective  helped  us  refine  our  elicitation  approach,  demonstrated  the  complexity 
of  populating  a  BBN  at  scale,  and  illuminated  the  need  for  calibrating  teams  of  experts,  not 
just  individuals.  Remaining  work  involves  obtaining  a  review  of  our  decisions  about 
connecting  the  BBN  to  cost  models  such  as  COCOMO  and  SEER. 

Conclusion 

Extensive  cost  overruns  have  been  endemic  in  defense  programs  for  many  years.  A 
significant  part  of  the  problem  is  that  the  information  used  for  cost  estimates  of 
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unprecedented  systems  must  rely  heavily  on  expert  judgments.  When  done  early  in  the 
system’s  life  cycle,  the  estimate  is  based  only  on  the  concept  and  incorporates  much 
uncertainty  as  to  how  that  concept  will  be  developed  into  a  fully  deployed  operational 
system.  QUELCE  aims  to  reduce  the  adverse  effects  of  that  uncertainty.  Important  program 
change  drivers  and  the  dependencies  among  them  that  may  not  otherwise  be  considered  in 
forming  estimates  are  made  explicit  to  improve  their  realism  and  accuracy.  The  basis  of  an 
estimate  is  documented  explicitly,  which  facilitates  updating  the  estimate  during  program 
execution  and  helps  others  to  make  informed  judgments  about  their  accuracy.  Variations  in 
the  range  of  possible  states  of  the  program  change  drivers  that  may  occur  under  different 
likely  scenarios  are  explicitly  considered.  The  use  of  probabilistic  methods  combining 
Bayesian  belief  systems  and  Monte  Carlo  simulation  will  ultimately  place  the  cost  estimates 
within  a  more  realistic  range  of  uncertainty. 
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Abstract 

Cost  estimates  and  other  analyses  for  acquisition  decisions  should  incorporate  fully-burdened 
costs  of  the  required  commodities  in  the  relevant  planning  scenarios.  In  addition  to  other 
widely  recognized  challenges  associated  with  estimating  fully-burdened  costs  of  supply, 
standard  approaches  systematically  produce  underestimates  for  self-sustaining  logistics 
networks.  The  disparity  is  especially  pronounced  when  multiple  commodities  consumed  by 
logistics  activities  are  not  locally  available.  This  work  develops  a  model  for  estimating 
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resource  demands  and  overall  cost  associated  with  self-sustaining  logistics  networks,  which 

can  then  be  applied  to  specific  examples. 

Introduction 

Analysis  supporting  acquisition  decisions  requires  the  calculation  of  fully-burdened 
costs  of  resources  consumed  by  the  systems  being  considered.  This  requires  an 
assessment  of  planning  scenarios  under  which  the  systems  may  be  operated.  In  many 
cases,  an  important  part  of  a  planning  scenario  is  the  logistics  network  that  supports  the 
system.  However,  a  Defense  Science  Board  (DSB)  task  force  was  “unable  to  identify  any 
case  where  the  logistics  reductions  or  deployment  and  sustainment  enhancements 
achievable  from  improvements  in  platform  efficiency  were  quantitatively  included  as 
capability  improvements  and  factored  into  trade-off  decisions”  (Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)],  2008).  This  work  is 
part  of  an  effort  to  allow  such  factors  to  be  included  in  a  quantitative  analysis  supporting 
acquisition  decisions. 

We  describe  a  logistics  network  as  “self-sustaining”  if  one  or  more  commodities 
consumed  by  the  logistics  activities  are  not  locally  available  and  must  therefore  be  supplied 
via  the  network  itself.  These  types  of  networks  are  common  for  operations  in  undeveloped 
or  disaster-impacted  regions.  The  costs  associated  with  self-sustaining  logistics  networks 
are  significantly  higher  than  those  of  traditional  logistics  networks.  Thus,  traditional 
approaches  to  cost  estimation  for  acquisition  decisions  tend  to  underestimate  actual  costs  of 
operating  systems  in  such  environments.  It  is  likely  that  the  implications  of  the  findings  of  the 
DSB  task  force  are  even  more  pronounced  when  the  additional  factor  of  self-sustainment  is 
considered.  The  purpose  of  this  work  is  to  build  a  framework  for  estimating  fully-burdened 
cost  of  supply  (FBCS)  in  self-sustaining  logistics  networks. 

Previous  work  has  identified  the  existence  of  a  “multiplier  effect”  for  fuel  in  multi¬ 
stage  self-sustaining  networks.  If  the  warfighter  requires  X  gallons  of  fuel,  some  proportion 
ofXis  consumed  by  the  preceding  stage  of  the  network,  and  thus  some  larger  amount  X+A 
is  required  at  the  start  of  this  stage.  The  stage  preceding  that  one  will  in  turn  consume  some 
proportion  of  X+A,  resulting  in  an  even  larger  requirement.  This  process  continues  all  the 
way  back  to  the  beginning  of  the  network,  where  the  fuel  requirement  may  be  substantially 
greater  than  X  gallons,  depending  on  characteristics  of  the  network.  For  more  details  on  the 
multiplier  effect,  see  Dubbs  (2011);  Regnier,  Simon,  and  Nussbaum  (2012);  and  Regnier, 
Simon,  Nussbaum,  and  Whitney  (2013). 

The  multiplier  effect  is  even  more  pronounced  when  multiple  resources  consumed  by 
the  logistics  activities  are  not  locally  available.  For  example,  consider  a  network  in  which 
both  fuel  and  water  must  be  supplied  via  the  network  itself.  If  an  additional  1,000  gallons  of 
fuel  are  needed  at  the  third  stage  of  the  network,  this  may  require  an  extra  convoy  on  the 
second  stage.  The  extra  convoy  will  not  only  consume  additional  fuel,  but  the  additional 
personnel  involved  will  consume  water  as  well.  Thus,  the  additional  fuel  requirement 
increases  the  requirements  for  both  fuel  and  water  at  earlier  stages  of  the  network.  A 
notional  illustration  of  this  phenomenon  is  shown  in  Figure  1.  These  interactions  are  not 
trivial  and  become  larger  and  more  complex  if  many  different  commodities  must  be 
transported  through  the  logistics  network. 
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Figure  1.  Illustration  of  the  Multiplier  Effects  and  Interactions  Between  Fuel  Demand  and 

Water  Demand  in  a  Self-Sustaining  Supply  Network 


Table  1,  reproduced  from  Regnieret  al.  (2013),  illustrates  the  single-commodity  fuel 
multiplier.  In  this  example,  a  total  of  1,794  gallons  of  fuel  are  required  at  the  beginning  of 
Stage  1  in  order  to  transport  and  deliver  1 ,000  gallons  of  fuel  to  the  end  of  Stage  3.  Table  2 
shows  an  example  which  is  similar  to  that  of  Table  1 ,  except  that  it  includes  two 
commodities.  In  this  example,  a  total  of  1,000  gallons  of  supply — fuel  and  water — are 
delivered  to  the  end  of  Stage  3.  The  fuel  requirements  of  each  stage  (as  a  percentage  of  the 
supply  delivered)  are  unchanged,  and  the  water  requirements  (as  a  percentage  of  supply 
delivered)  are  90%  lower  than  the  fuel  requirements.  However,  transporting  either  fuel  or 
water  requires  consumption  of  both  commodities.  The  total  amount  of  fuel  and  water 
required  in  this  example  is  1,890  gallons,  an  increase  of  5.4%  relative  to  the  Table  1 
example,  although  the  per-stage  water  requirements  do  not  exceed  3%.  This  demonstrates 
the  impact  of  multiple  commodities  on  the  operating  costs  of  self-sustaining  supply 
networks. 


Table  1.  Example  of  a  Single-Commodity  Self-Sustaining  Supply  Network 

(reproduced  from  Regnier  et  al.,  2013) 


Fuel 

Delivered 

(gal) 

Fuel 

C  onsumption 
(%  of  delivered) 

Operating  Costs 

Non-Fuel  Fuel 

Total 

Total  Operating 
Costs  per 
Gallon 

Delivered 

Stage  1 

1560 

15% 

$3,120 

$538 

$3,658 

$2.35 

Stage  2 

1200 

30% 

$2,400 

$828 

$3,228 

$2.69 

Stage  3 

1000 

20% 

$2,000 

$460 

$2,460 

$2.46 

Total 

1794 

79% 

$7,520 

$1,826 

$9,346 
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Table  2.  Example  of  a  Two-Commodity  Self-Sustaining  Supply  Network 


Fuel 

Delivered 

(gal) 

Fuel 

Consumption 
(%  of  delivered) 

Water  Delivered 

(gal) 

Water 

Consumption 
(%  of  delivered) 

Total  Resources 

Delivered 

Non- 

Resource 

Operating  Costs 

Resource 

Total 

Total  Operating 
Costs  per 
Gallon 

Delivered 

Stage  1 

1466 

15% 

157 

1.5% 

1,623 

$3,732 

$616 

$4,348 

$2.97 

Stage  2 

1100 

30% 

120 

3% 

1,220 

$2,806 

$926 

$3,732 

$3.39 

Stage  3 

900 

20% 

100 

2% 

1,000 

$2,300 

$506 

$2,806 

$3.12 

Total 

1,890 

$8,838 

$2,048 

$10,886 

Note.  The  model  in  this  example  includes  consumption  of  both  fuel  and  water. 


As  we  have  noted  previously,  analyses  of  costs  and  requirements  should  not  be 
conducted  independently  for  each  stage,  because  the  resulting  quantities  are  not  additive. 
Similarly,  analyses  of  costs  and  requirements  should  not  be  conducted  independently  for 
each  commodity,  because  these  resulting  quantities  are  not  additive  either.  Capturing  the 
cross-commodity  impacts  for  many  commodities  is  difficult  to  do  by  estimating  unit  costs  of 
delivery  on  each  stage,  especially  as  the  number  of  commodities  supplied  via  the  supply 
network  itself  increases. 

We  have  previously  used  input-output  analysis  to  estimate  fully-burdened  costs  of 
fuel  (FBCF)  in  single-commodity  supply  networks,  which  can  be  found  in  the  references 
given  earlier  in  this  section.  This  approach  can  be  extended  to  include  the  types  of  cross¬ 
commodity  impacts  used  in  the  example  shown  in  Table  2.  The  general  input-output 
approach  was  developed  by  Leontief  (1970,  1986).  Based  on  the  previous  application  of 
input-output  analysis  to  FBCF,  this  work  expands  the  approach  to  estimate  FBCS  given  any 
number  of  commodities  in  a  self-sustaining  logistics  network. 

Model 


The  multi-commodity  FBCS  model  is  presented  in  this  section.  Further  details  and 
derivations  of  results  were  given  by  Regnier  and  Simon  (2013).  The  model  examines  one 
individual  path  through  the  logistics  network.  Let  this  path  have  n  nodes.  We  refer  to  the 
stage  which  begins  at  node  /  and  ends  at  node  /'+ 1  as  stage  /;  the  path  has  n-1  stages.  We 
assume  there  are  m  different  commodities  transported  on  it,  indexed  by  c.  We  also  assume 
that  all  commodities  can  be  expressed  in  the  same  units,  whether  by  weight  or  by  volume. 
The  model  includes  the  following  parameters: 

xcn  -  amount  of  commodity  c  needed  at  the  destination  (exogenously  given 
requirement) 

x.  -  amount  of  commodity  c  required  at  node  / 

m 

X i  -  total  requirement  at  node  / .  NoteX;.  =  ^ x . 

c= 1 

di  -  distance  of  stage  /'  (i.e.,  from  node  /  to  node  /+1) 

r?  -  amount  of  commodity  c  consumed  per  unit  distance  on  stage  / 

m 

-  total  consumption  per  unit  distance  on  stage  /.  Notei?,  = 

c= 1 

at  -  number  of  personnel  required  on  convoy  in  stage  / 

p.  -  average  speed  on  stage  /  (includes  time  spent  loading  and  unloading) 

wi  -  total  convoy  capacity  on  stage  /,  including  payload  plus  internal  fuel  tanks 


m  khsiikm 
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a .  -  amount  of  commodity  c  consumed  at  node  /  per  hour  of  labor  on  stage  / 

m 

4  -  total  consumption  at  node  /  per  hour  of  labor  on  stage  /.  Note  Ai  =  ^ a ;c 

c= 1 

4  -  operating  &  support  cost  per  unit  of  distance  for  the  convoy  on  stage  /  (i.e. , 

vehicle  depreciation,  maintenance  costs,  and  any  similar  costs  not  explicitly 
captured  in  consumption) 

yc  -  unit  cost  of  purchasing/producing  commodity  c  at  the  start  of  the  supply  chain 
(let  yL  represent  the  cost  of  labor) 

The  values  of  these  parameters  will,  of  course,  depend  on  the  particular  logistics 
network  being  analyzed.  Many  of  the  parameters  are  easily  obtainable  given  a  familiarity 
with  the  network.  For  example,  if  the  convoy  composition  for  a  stage  is  known,  several  of  the 
parameters  are  straightforward  to  compute. 

Analysis 

Two  intermediate  calculations  are  helpful  before  presenting  any  general  results.  The 
number  of  convoy  round-trips  Ki  required  on  stage  /  can  be  expressed  as 


K 

l 


X 


f+1 


vv;  -  2  d.R, 


(1) 


The  denominator  represents  the  total  amount  of  commodities  which  can  be  delivered  to 
node  /+ 1  by  the  convoy  on  one  round-trip.  (This  expression  is  an  approximation  because 
fractional  round-trips  are  impossible;  the  size  of  the  error  is  trivial  if  the  number  of  round-trips 
is  large.)  The  model  allows  for  replenishment  of  logistics  assets  within  a  stage — the  distance 
of  a  stage  is  not  constrained  by  the  internal  fuel  tank  of  a  transportation  asset,  for  example. 

It  will  also  be  helpful  to  compute  Lj :  the  number  of  labor  hours  required  per  convoy  round- 
trip  on  stage  /.  It  can  be  expressed  as 


Z.  =  a: 


2d, , 


1  Pi 


(2) 


Given  Kt  and  Li ,  it  is  possible  to  compute  requirements  for  each  commodity  at  each  node: 


C 

i+ 1 


+  2  K.d.rc 

i  i  i 

\ _ _ _ j 

v 

amount  of 
resource  c 
consumed 
in  transport  on 
stage  i 


+ 


amount  of 
resource  c 
consumed  to 
sustain  convoy 
personnel 
while  at  node  i 


(3) 


for  i  =  This  expression  is  recursive;  the  requirements  at  a  given  node  are  a 

function  of  the  requirements  at  the  following  node.  Given  these  relationships  between 
requirements,  the  total  FBCS  for  this  path  through  the  supply  network  is  given  by 

m  n— 1  n— 1 

Z xi  x  +  2Z dPK'  +>’/.Z LiKi  ■  (4) 

C= 1  1=1  1=1 
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At  the  operational  planning  level,  the  above  calculations  are  unlikely  to  be 
managerially  relevant.  Many  costs  included  in  (e.g.,  acquisition  costs)  are  sunk.  Even 

variable  costs  such  as  labor  often  cannot  be  influenced  by  operational  logistics  decisions  in 
theater.  Labor  and  other  resources  may  be  diverted  from  other  tasks  to  logistics  support, 
however.  More  important,  at  the  strategic  level,  all  costs  are  variable — they  may  all  be 

influenced  by  decisions  that  affect  the  total  end  user  demand  ( xcn )  and  efficiency  of  logistics 

(a-  ,r.t  and  (/>.). 


The  total  FBCS  estimate  is  intended  to  be  used  in  strategic-level  assessments  of  the 
magnitude  of  costs  of  supply  to  a  particular  area.  However,  being  able  to  compute  the 
overall  FBCS  also  allows  us  to  answer  more  specific  questions  about  the  impacts  of 
acquisition  decisions  on  total  costs. 

To  build  a  framework  for  answering  the  types  of  questions  relevant  to  acquisition 
decisions,  we  will  introduce  several  concepts  analogous  to  the  fuel  multiplier  in  a  single¬ 
commodity  network.  One  such  concept  is  a  stage  multiplier  A,. ,  which  is  expressed  for  any 

stage  /  as 


I  |  Id.R  +  LA! 
wt  -  2djRj 


(5) 


The  stage  multiplier  shows  the  increase  in  total  requirement  at  node  /  per  unit  of 
increase  in  the  total  requirement  at  node  /'+ 1 .  Another  helpful  concept  is  a  cross-commodity 

factor ,  which  is  expressed  for  any  commodity  c  and  stage  /  as 


Xt  =- 


2drc  +  acjLi 
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(6) 


The  cross-commodity  factor  shows  the  increase  in  the  required  amount  of 
commodity  c  at  node  /  per  unit  of  increase  in  the  amount  of  a  different  commodity  required  at 
node  /+ 1 . 

Based  on  Equations  5  and  6,  it  is  possible  to  construct  a  factor  which  captures  such 
relationships  across  multiple  stages,  denoted  as  ^  ■  This  factor  is  expressed  as 


r-i 


Xu 


7-1 


= i  n  a, 
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(7) 


for  any  commodity  c  and  nodes  /  and  j,  i  <  j.  It  indicates  the  increase  in  the  amount  of 
commodity  c  required  at  node  /  per  unit  increase  in  the  amount  of  a  different  commodity 
required  at  node  j.  Note  that  Equation  7  expresses  the  relationship  between  the 
consumption  of  a  commodity  at  a  given  node  with  the  requirement  of  any  other  commodity 
at  any  other  node.  In  particular,  when  /  =  1 ,  Equation  7  shows  the  additional  amount  of 
commodity  c  needed  at  the  beginning  of  the  supply  network,  which  can  be  used  to 
determine  the  impact  on  total  cost.  Further  details  and  mathematical  results  were  given  by 
Regnierand  Simon  (2013). 

For  example,  consider  a  new  platform  which  decreases  the  warfighter’s  fuel 
requirement  by  A  gallons.  The  cost  savings  resulting  from  a  decrease  in  the  FBCS  would  be 
given  by 
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2>^a 

c= 1 

These  savings  are  in  addition  to  the  savings  achieved  as  a  result  of  not  consuming 
those  A  gallons  of  fuel  themselves,  which  would  be  equal  to  A  multiplied  by  the  per-gallon 
market  price  of  fuel  at  the  start  of  the  network. 

Discussion  and  Conclusions 

The  model  given  above  applies  to  any  mode  of  transport  and  can  model  multi-modal 
logistics  networks — for  example,  a  sea-based  stage,  followed  by  ground  transportation, 
followed  by  air  delivery.  However,  there  are  some  important  systematic  differences  by  mode. 
In  Regnier  et  al.  (2013),  we  provided  a  model  of  ground-based  transport,  in  which  each 
stage’s  resource  requirements  are  determined  by  the  distance  and  the  composition  of  a 
logistics  convoy.  This  model  is  single-commodity  but  nevertheless  highlights  the  fact  that 
land-based  stages  are  highly  sensitive  to  variations  in  terrain  and  infrastructure  that  are  less 
relevant  to  air  and  sea-based  stages.  Relative  to  sea-based  transport,  the  assumption  of  a 
large  number  of  round-trips  is  more  appropriate.  In  addition,  because  the  payload  of  each 
vehicle  is  much  smaller  than  the  payload  of  a  vessel,  convoy  composition  for  land  stages  is 
more  flexible  and  can  be  tailored  to  specific  commodity  requirement  distribution,  which 
supports  treating  commodities  as  interchangeable. 

Based  on  the  methods  in  this  work,  Hathorn  (2013)  developed  a  model  for  fully 
burdened  costs  of  supply  in  a  naval  supply  network  under  different  possible  threat 
scenarios.  The  complete  network  is  shown  in  Figure  2,  reproduced  from  Hathorn’s  paper. 
The  models  presented  previously  can  be  applied  to  any  individual  route  through  this  supply 
network.  In  Hathorn’s  model,  force  protection  is  an  important  consideration;  multiplier  effects 
and  interactions  between  commodities  are  much  more  significant  when  force  protection 
vehicles  are  required  in  addition  to  transportation  vehicles.  In  the  network  being  studied,  fuel 
is  the  commodity  that  has  by  far  the  highest  level  of  consumption — over  95%.  However, 
other  commodities  such  as  stores  and  ordnance  are  included  as  well. 
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(reproduced  from  Hathorn,  2013) 

Hathorn  also  demonstrated  that  the  FBCS  model  can  be  valuable  in  supporting  other 
types  of  decision  problems.  In  particular,  it  allows  for  route  selection  decisions  to  be  made 
with  more  complete  information  about  costs.  Hathorn  introduced  an  optimization  model  for 
route  selection  which  determines  how  to  provide  a  given  amount  of  supply  to  a  specified 
location  at  minimum  (fully-burdened)  cost.  Constraints  may  be  added  to  the  optimization 
model  based  on  the  current  environment;  for  example,  there  may  be  scenarios  in  which 
certain  arcs  in  the  network  are  unavailable. 

As  an  example,  Hathorn  analyzed  a  supply  route  from  San  Diego  to  the  Spratly 
Islands.  An  illustration  of  this  supply  route  is  shown  in  Figure  3,  reproduced  from  Hathorn 
(2013).  Depending  on  threat  level  and  convoy  composition,  the  total  cost  per  short  ton  of 
supply  delivered  to  the  destination  ranges  from  $1,638.70  to  $3,144.47.  When  developing 
planning  scenarios  to  support  decision-making,  it  is  important  to  consider  the  possibility  of 
both  high-threat  and  low-threat  environments,  as  the  associated  fully-burdened  costs  of 
supply  can  be  extremely  different. 


Figure  3.  A  Possible  Supply  Route  From  San  Diego  to  the  Spratly  Islands 

(reproduced  from  Hathorn,  2013) 

Hathorn’s  work  highlights  the  importance  of  considering  ammunition  requirements  of 
the  logistics  network  in  a  high-threat  environment.  Consumption  of  ammunition  during  force- 
protection  may  be  considered  a  requirement — rather  than  a  choice — driven  by  the  threat 
and  thus  might  reasonably  be  modeled  using  planning  factors.  However,  there  could  be  a 
very  wide  range  of  assumptions  about  the  appropriate  ammunition  consumption  rate 

(parameter  for  c  =  ammunition).  In  addition,  ammunition  requires  specialized 

transportation  assets,  and  different  kinds  of  ammunition  have  very  different  requirements 
(cruise  missiles  vs.  anti-submarine  torpedoes).  Modeling  their  demand  by  the  warfighter  and 
logistics  ammunition  requirements  in  planning  scenarios  during  acquisition  is  an  important 
but  challenging  problem. 
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Abstract 

The  approach  presented  here  combines  techniques  from  multidisciplinary  design  optimization 
and  operations  research  to  improve  energy  efficiency-related  defense  acquisition  decisions. 
The  work  focuses  upon  the  acquisition  of  new  aircraft  for  the  U.S.  Air  Force  Air  Mobility 
Command  missions.  Air  Mobility  Command  is  the  largest  consumer  of  fuel  in  the  Department 
of  Defense,  making  this  a  relevant  example  application.  The  approach  here  builds  upon 
previous  efforts  that  examined  fleet-level  acquisition  decisions  for  commercial  airline-related 
problems,  so  the  paper  describes  changes  necessary  to  use  the  problem  decomposition 
strategy  of  the  previous  applications  in  the  context  of  Air  Mobility  Command.  With  many  of 
these  changes  made,  the  approach  is  used  to  simultaneously  select  requirements  for  a  new 
cargo  aircraft;  predict  size,  weight,  and  performance  of  that  new  aircraft;  and  also  allocate  the 
new  aircraft  along  with  existing  aircraft.  The  fuel  efficiency  of  the  resulting  fleet  provides  a 
metric  for  comparison.  The  approach,  with  the  abstractions  and  assumptions  used, 
successfully  provides  a  description  of  a  new  cargo  aircraft  that  impacts  fleet-level  metrics. 
Results  in  this  study  consider  a  simplistic  three-route  network  and  two  larger  networks,  all 
informed  by  actual  Air  Mobility  Command  data  captured  by  the  Global  Air  Transportation 
Execution  System. 

Introduction 

The  Energy  Efficiency  Starts  with  the  Acquisition  Process  factsheet  (DUSD[AT&L], 
2012)  states,  “Neither  current  requirements  or  acquisition  processes  accurately  explore 
tradeoff  opportunities  using  fuel  as  an  independent  variable.”  The  factsheet  also  states, 
“Current  processes  undervalue  technologies  with  the  potential  to  improve  energy  efficiency.” 
Studies  conducted  by  the  Institute  for  Defense  Analyses,  the  Defense  Science  Board, 
Energy  Security  Task  Force,  and  JASON  have  all  alluded  to  the  significant  risk  and 
operational  constraints  that  energy  efficiency  issues  pose  on  military  operational  flexibility. 
The  consumption  and  transport  of  fuel  across  a  combat  theater,  throughout  the  life  cycle  of 
operational  systems,  poses  significant  operational  risk,  strategic  vulnerability,  and  increased 
monetary  cost  in  supporting  forward-force  assets.  Additionally,  increasing  fuel  consumption 
shifts  focus  to  the  acquisition  of  an  increasing  number  of  “tail  units”  in  maintaining  forward- 
force  assets.  Aviation  fuel  contributes  the  largest  percentage  of  energy  consumption  in  the 
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Department  of  Defense  (DoD),  with  the  Air  Mobility  Command  (AMC)  being  the  single 
largest  consumer  (Allardice,  2012).  This  makes  an  air  mobility-related  application  relevant 
for  the  current  research  effort. 

AMC  is  a  branch  of  the  United  States  Air  Force  that  is  responsible  for  a  wide  range 
of  airlift  missions  that  span  its  global  theater  of  operations.  AMC’s  mission  profile  mainly 
consists  of  worldwide  cargo  and  passenger  transport,  air  refueling,  and  aeromedical 
evacuation.  AMC  also  provides  transports  for  humanitarian  supplies  for  major  natural 
disaster  around  the  world.  Platforms  in  operation  include  C-5  Galaxy,  C-17  Globemaster  III 
for  long  range  strategic  missions,  C-130  Hercules  for  tactical  missions,  KC-135  Stratotanker, 
and  KC-10  Extender  for  aerial  refueling  missions,  and  various  VIP  transport  platforms 
including  Air  Force  One.  AMC  also  utilizes  Civil  Reserve  Air  Fleet  contractually  committed 
from  U.S.  airlines  (Air  Mobility  Command,  2013). 

The  complex  logistics  involved  in  the  transportation  of  various  cargos  across  its 
service  network  requires  effective  deployment  of  its  fleet  of  cargo  aircraft  in  meeting  daily 
cargo  delivery  requirements,  while  minimizing  fuel  consumption  and  subsequent  costs. 
These  fuel  costs  are  naturally  driven  by  the  choice  of  aircraft  design  and  individual  flight  legs 
flown  by  the  AMC  fleet,  in  meeting  cargo  obligations  within  a  prescribed  schedule 
timeframe.  The  identification  of  cost-saving  measures  in  minimizing  fleet-wide  fuel 
consumption  is  thus  intuitively  tied  into  the  design  of  the  aircraft  itself,  and  the  structure  of 
the  routes  flown.  However,  the  characteristics  of  aircraft  flown  dictate  the  kind  of  network 
that  the  fleet  can  serve,  thus  making  it  a  closely  coupled  problem. 

The  objective  of  this  work  is  to  provide  a  decision-support  framework  that  assists 
acquisition  practitioners  in  identifying  optimal  characteristics  of  new  assets  (here,  aircraft) 
that  can  minimize  fuel  dependency  of  the  entire  system  architecture  in  which  they  serve 
(here,  the  fleet  of  cargo  aircraft).  This  context  is  driven  by  the  coupled  nature  that  an  aircraft 
design  has  on  fleet  operations.  The  framework  in  this  paper  provides  a  process  that  can 
examine  how  acquisition  (and  pre-acquisition)  decisions  describing  the  requirements  for  a 
new  aircraft  might  be  made  to  directly  reduce  fleet-level  fuel  usage/cost,  considering  the 
operational  network  and  other  existing  assets  along  with  the  potential  new  (or  modified) 
platform.  Consideration  of  the  aircraft  design  and  fleet  allocation  problems  simultaneously 
presents  many  decision  variables — a  condition  where  the  size  of  the  problem  rapidly 
exceeds  the  mental  capability  of  the  designer.  Hence,  a  computational  approach  becomes 
necessary  to  address  the  complexities  associated  with  the  coupled  problem.  This  research 
will  advance  the  knowledge  on  how  to  perform  trade-offs  with  fleet-level  fuel  consumption  as 
one  of  the  quantities  of  interest  and  will  enhance  understanding  about  what  features  this 
kind  of  process  should  entail. 

Problem  Statement 

Previously  research  at  Purdue  University  has  used  decomposition  strategies  that 
allow  a  direct  connection  between  the  design  of  a  new  system  (here,  an  aircraft)  and  its 
operations  along  with  other  existing  systems  (here,  a  fleet  of  aircraft).  The  result  is  an 
approach  that  can  maximize  or  minimize  a  fleet-level  objective  function  by  searching  for  a 
set  of  decision  variables  that  describe  the  new  system  design  and  describe  the  allocation  of 
the  new  and  existing  systems  to  perform  operational  missions.  While  a  single,  monolithic 
problem  statement  can  reflect  this  kind  of  problem,  solution  of  the  resulting  mixed-integer, 
non-linear  programming  problem  (MINLP)  is  difficult,  if  not  impossible.  The  decomposition 
strategy  breaks  down  the  computational  complexity  of  the  decision  space  into  a  series  of 
smaller  subproblems  controlled  by  a  top-level  problem.  The  decomposition  approach 
addresses  the  issue  of  tractability,  of  solving  a  monolithic,  mixed  discrete  non-linear 
programming  problem,  and  has  yielded  better  “design  solutions”  across  a  set  of  aviation 
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applications  including  commercial  airlines,  fractional  management  companies,  and  air  taxi 
services  (Mane  &  Crossley,  2006,  2012;  Mane,  Crossley,  &  Nusawardhana,  2007).The 
motivation  of  these  prior  works  in  identifying  cost-  and  fuel-saving  characteristics  of  a  new, 
yet-to-be-acquired  aircraft  bears  great  similarity  to  the  U.S.  Air  Force  Air  Mobility  Command 
(AMC)  problem.  This  paper  presents  a  process  that  allows  investigation  of  trade-offs 
between  fleet-level  fuel  usage,  performance  metrics,  and  acquisition  alternatives  for  a 
conceptual  problem  that  resembles  missions  of  the  AMC. 

AMC’s  automated  air  transportation  management  system,  Global  Air  Transportation 
Execution  System  (GATES),  is  managed  by  USTRANSCOM  and  has  very  detailed 
information  on  palletized  cargo  and  personnel  transported  by  the  AMC  fleet.  Cargo 
transported  by  the  strategic  fleet,  consisting  of  C-5  and  C-17  aircraft,  and  the  Boeing  747 
Freighter  (747-F)  from  the  Civil  Reserve  Air  Fleet  (CRAF)  for  long-range  missions,  are 
considered  as  a  representative  measure  of  typical  cargo  flow  on  the  AMC  service  network. 
Each  data  item  entered  in  “GATES  Pallet  data”  represents  cargo  on  a  pallet  or  a  pallet-train 
that  was  transported.  Each  pallet  data  entry  item  has  detailed  information  about  the  pallet, 
such  as  pallet  gross  weight,  departure  date  and  time,  arrival  date  and  time,  mission 
distribution  system  (MDS),  tail  number,  aerial  port  of  embarkation  (APOE),  aerial  port  of 
debarkation  (APOD),  pallet  volume,  pallet  configuration,  and  so  forth.  These  data  enable  the 
reconstruction  of  the  route  network,  pallet  demand  characteristics,  and  existing  fleet  size  for 
our  allocation  problem. 

In  this  paper,  the  following  assumptions  are  made  on  operations  of  the  fleet,  based 
on  the  available  dataset: 

In  this  paper,  the  following  assumptions  are  made  on  operations  of  the  fleet,  based 
on  the  available  dataset: 

1 .  The  filtered  route  network  from  the  GATES  dataset  is  representative  of  all 
AMC  cargo  operations. 

a.  Demand  for  the  subset  served  by  C-5,  C-17,  and  747-F  (75%  of  all 
pallets  in  the  GATES  dataset) 

b.  Fixed  density  and  dimension  of  the  pallet,  representing  the  463L  pallet 
type 

2.  The  aircraft  fleet  consists  of  only  the  C-5,  C-1 7,  and  747-F.  The  model  is 
indifferent  to  variants  of  these  aircraft  types. 

3.  Aircraft  operate  on  a  round  trip  between  each  base  pair  to  avoid  time-of-day 
scheduling  issues  and  the  need  for  flow  balance  constraints.  A  round  trip 
consists  of  a  trip  from  the  hub  airport  to  the  outlying  base  airport  and  a  return 
trip  from  the  outlying  base  airport  to  the  hub  airport.  This  assumption  played 
an  important  role  in  simplifying  the  previous  work  for  passenger  airline 
problems  and  was  reasonable  for  scheduled  passenger  service.  This 
assumption  does  not  appear  as  acceptable  for  AMC  cargo  operations; 
however,  work  to  date  has  not  removed  this  assumption. 

Example  Baseline  Three-Route  Problem 

We  motivate  our  study  with  a  very  simple,  illustrative  “baseline”  problem  for  AMC 
operations.  In  this  scenario,  a  representative  route  network,  consisting  of  three  routes  with 
one  shared  base,  is  drawn  from  the  GATES  dataset  for  2006.  A  schematic  of  the  sample 
problem  network  appears  in  Figure  1.  The  three  aircraft  operated  on  these  routes  are  the  C- 
5,  C-17,  and  the  Boeing  747-F  (the  latter  of  which  is  assumed  to  be  operated  as  a  chartered 
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aircraft).  In  this  simplified  problem,  we  make  the  assumption  that  the  aircraft  operates  on  a 
round  trip  basis  and  that  the  amount  of  palletized  cargo  between  each  base  and  the  Hub 
base  is  symmetrical.  Route  1  has  a  range  of  2,495  nautical  miles  with  2,775  pallets 
transported  each  way  in  one  year.  Route  2  has  a  range  of  325  nautical  miles  with  2,1 15 
pallets  transported.  Route  3  has  a  range  of  1,101  nautical  miles  with  2,199  pallets 
transported  in  2006.  The  maximum  distance  of  the  three  chosen  routes  is  2,495  nautical 
miles,  which  allows  all  three  types  of  current  strategic  airlift  aircraft  to  provide  service  on 
these  routes  without  refueling.  The  intent  is  to  allocate  aircraft  to  the  three  routes  to  satisfy 
all  cargo  demand. 


Figure  1 .  Schematic  of  Three-Route  Allocation  Problem 


Aircraft  Sizing  and  Costs 

When  determining  which  aircraft  to  allocate  to  the  network  routes,  the  problem 
formulation  will  require  estimates  of  the  cost,  block  time,  and  fuel  consumed  by  each  aircraft 
type  in  the  fleet.  A  Purdue  in-house  aircraft  sizing  code,  written  in  MATLAB,  provides  these 
estimates.  Jane’s  Aircraft  database  (Jackson,  Peacock,  &  Munson,  2004)  provided  the  input 
parameters  for  the  three  existing  aircraft  types  (C-5,  C-17,  747-F)  used  in  this  study,  as 
shown  in  Table  1 . 


Table  1.  Existing  Aircraft  Characteristics 


Parameter 

C-5 

C-17 

747-F 

Range  (nmi) 

2,982 

2,420 

4,445 

Pallet  Capacity 

36 

18 

29 

WlSj\ b/fPj 

135.48 

161.84 

137.34 

TIW 

0.205 

0.263 

0.286 

AR 

7.75 

7.2 

7.7 
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Direct  operating  cost  (DOC)  estimates  for  commercial  aircraft  usually  include  fuel 
costs,  crew  costs,  maintenance,  depreciation,  and  insurance.  DOC  estimates  are  also 
dependent  on  the  payload,  route  distance,  empty  weight,  landing  weight,  and  take-off  gross 
weight.  While  AMC  does  not  have  the  same  operating  cost  structure,  the  problem 
formulation  here  started  using  total  fleet  operating  cost  as  the  objective  function.  Because 
cost-estimating  relationships  exist  for  commercial  aircraft,  the  AMC  formulation  uses  these 
estimators,  even  if  they  may  not  directly  match  the  costs  for  AMC  operations.  The  trip  DOC 
of  each  nominally  loaded  (based  on  typical  loaded  operations)  aircraft  type,  for  each  route, 
appears  in  Table  2. 


Table  2.  Aircraft  Operating  Costs  of  Flight  for  Each  Route 


Aircraft  Type 

Route  1  Cost 

Route  2  Cost 

Route  3  Cost 

Aircraft  1  (C-5) 

$130,503 

$54,752 

$81,671 

Aircraft  2  (C-1 7) 

$107,299 

$43,858 

$66,098 

Aircraft  3  (747F) 

$141,124 

$62,691 

$90,358 

Figure  2  shows  a  typical  mission  profile  used  for  the  aircraft  sizing  and  operating 
missions.  To  compute  the  fuel  weight  necessary  for  flying  the  route  distance,  the  fuel 
required  for  each  mission  segment  is  computed  and  aggregated.  The  fuel  weight  fractions 
for  the  different  mission  segments  such  as  warm-up  and  take-off,  climb,  landing  and  taxi, 
and  reserves  are  based  on  empirical  data  presented  in  Raymer’s  textbook  (2006).  To 
compute  the  fuel  weight  fractions  for  the  cruise  and  loiter  mission  segments,  the  Breguet 
range  and  endurance  equations  are  used.  The  descent  segment  uses  a  no-range  credit 
assumption.  The  reserve  fuel  fraction  is  assumed  to  be  6%,  which  also  accounts  for  a  small 
amount  of  trapped  and  unusable  fuel. 


The  payload-range  curves  for  the  existing  aircraft  fleet,  depicted  in  Figure  3,  indicate 
the  maximum  payload  carrying  capacity  of  the  aircraft  as  a  function  of  the  distance  flown  by 
the  aircraft.  The  payload-range  curves  for  the  existing  fleet  are  constructed  by  using 
piecewise  linear  interpolation  between  specified  points  from  charts  presented  in  Baker, 
Morton,  Rosenthal,  and  Williams  (2002). 
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Figure  3.  Payload  Range  Curves  for  Existing  Fleet 


Traditional  Aircraft  Allocation  Problem 

Using  the  information  provided  on  the  aircraft  flight  costs  (including  fuel  costs),  the 
objective  for  the  allocation  problem  seeks  to  minimize  fleet-level  DOC  by  allocating  the 
available  fleet  to  the  three  routes.  Cost  coefficients  from  Table  2  are  used  in  the  formulation 
of  the  following  mathematical  programming  problem.  Mathematical  programs  have  two 
important  aspects  of  formulation;  the  objective  function  that  reflects  the  metric  being 
minimized/maximized  and  constraints  that  reflect  resource  constraints  to  the  problem.  The 
decision  variables  are  the  variables  of  interest  that  can  be  manipulated  to  optimize  the 
objective.  The  allocation  problem  statement  is  as  follows: 


3 

minimize  Fleet  DOC  =  ^  [CAixA 

i=l  AeC-5, 

C-17,747-F 

]' 

(1) 

subject  to 

YjxAi<BAi  A  -  C-5,C-17,747-F 

1=1 

(trip  limits/aircraft  count) 

(2) 

Z  CaP  uXA,  ^  Ci 

AeC-5,  C-17, 

5-747 

(capacity) 

(3) 

xAi  6  int  ,  xM  >  0 

(4) 

In  the  case  of  the  traditional  aircraft  allocation  problem,  the  objective  function  in 
Equation  1  seeks  to  minimize  the  fleet  DOC.  The  decision  variable  is  given  by  xAi  (with 
subscripts  for  aircraft  type  and  route)  and  is  an  integer,  making  the  allocation  problem  an 
integer  programming  problem.  The  total  fleet  DOC  is  the  sum  of  costs  associated  with  the 
number  of  round  trips  an  aircraft  of  type  A  flies  on  route  /.  The  constraints  expressed  in 
Equations  2  and  3  are  the  aircraft  trip  limit  and  cargo  capacity  limits  on  each  route  (/).  The 
trip  limit  constraints  account  for  the  number  of  aircraft  available;  the  limiting  values  for 
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number  of  trips  operated  by  a  given  aircraft  type  in  one  year  are  based  upon  information 
from  the  GATES  data. 

AMC  Fleet  Allocation  Including  Design  of  New  Aircraft 

Here,  we  extend  the  AMC  aircraft  allocation  problem,  to  consider  the  potential  addition  of 
a  new,  yet-to-be-designed  aircraft,  and  its  impact  on  fleet-wide  operating  costs  and  fuel 
consumption.  The  optimization  problem  now  needs  to  consider  the  aircraft  costs  of  the  new 
aircraft  as  a  function  of  the  variables  describing  the  new  aircraft.  The  monolithic  optimization 
problem  simultaneously  considers  the  aircraft  design  and  allocation  of  the  fleet’s  aircraft  to 
meet  demand  obligations  and  is  given  by  the  following  equations. 

Minimize 


Fleet  DOC 


Subject  to 


3 

I 

1 

M 

o 

_ I 

+  Cs  (Pallet X,(AR)X  ,(W / S) x  ,(T /W) x  ) 

i=l 

AeC-5, 

C-\7 ,747-F 

3 

^xAi<BAi  A  =  C-5,C-17,747-F,  X  (trip  limits/aircraft  count) 

(=i 


X  CaP.4ixAi  -  Cj  (capacity) 

AeC-5,  C- 17, 

747  -F,  X 


Sm(Palletx,(AR) x ,(W / S) x ,(T /W) v)< D  (aircraft  take-off  distance) 

6  <  Pallet x  <36 

6.0<(AR)v<9.5 

65<(W/S)X<161 


(5) 

(6) 

(7) 

(8) 
(9) 

(10) 

(11) 


0.18<(r/r)x<0.35  (12) 

x Aj, Pallet x  eint,  xAi  >0  ("13) 

Equation  5  is  the  objective  function  that  seeks  to  minimize  the  fleet’s  DOC.  This 
equation  can  be  modified  for  different  studies  as  alternate  objectives,  such  as  directly 
minimizing  fuel  consumption,  and  so  forth,  are  considered.  Equation  6  preserves  the  aircraft 
trip  limits  for  a  typical  year  from  values  calculated  from  existing  flight  data;  this  represents 
utilization  rate.  Equation  7  ensures  sufficient  pallet  capacity  for  cargo  traveling  on  route  /'. 
Equations  8-13  limit  the  aircraft  design  based  on  minimum  take-off  distance  to  ensure  that 
the  new  aircraft  can  operate  at  bases  in  the  network.  The  continuous  design  variables 
describing  the  new  aircraft  area  were  limited  to  remain  near  the  range  of  values  associated 
with  current  cargo  aircraft.  As  in  the  “traditional  allocation”  problem,  the  number  of  trips  of 

each  aircraft  type,  XAl  are  integers.  The  coupling  of  the  fleet  allocation  (integer 

programming)  with  the  aircraft  design  (non-linear  programming)  makes  the  resource 
allocation  problem  a  mixed-integer,  non-linear  (MINLP)  problem.  MINLP  problems  are 
sometimes  impossible  to  solve  for  even  moderate-sized  problems.  However,  we  adopt  a 
Multidisciplinary  Design  Optimization  (MDO;  inspired  subspace  decomposition  approach 
from  prior  literature;  Mane  et  al.,  2007)  that  breaks  the  monolithic  MINLP  problem  of 
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Equations  5-13  into  a  coordinated  sequence  of  more  tractable  problems,  as  depicted  in 
Figure  4. 

Volumetric  load  factor  is  a  measure  introduced  as  the  ratio  of  the  number  of  pallets 
carried  to  the  maximum  pallet  capacity  of  the  aircraft  type.  As  the  density  of  cargo  varies  by 
missions,  the  average  weight  of  a  pallet  is  calculated  from  the  route  data  and  used  as  the 
pallet  weight  for  the  entire  route.  The  volumetric  load  factor  of  the  new  aircraft  is  assumed  to 
be  the  average  of  the  volumetric  load  factor  of  the  existing  aircraft  types  on  that  route. 

^  Load  factor^. 

Load  factor^.  = - -  (14) 

#  of  aircraft  type  operated  on  route  i 

The  volumetric  load  factor  formulation,  together  with  average  weight  of  the  pallet 
calculated  implicitly,  assumes  that  the  new  aircraft  would  be  operationally  utilized  in  a  similar 
manner  to  existing  aircraft.  The  GATES  dataset  is  limited  to  the  AMC  operations  involving 
palletized  cargo.  The  design  of  the  new  aircraft  is  strongly  influenced  by  the  operational 
characteristics  of  the  existing  AMC  fleet  and  the  AMC  route  network  as  described  in  the 
GATES  dataset.  However,  existing  aircraft  in  the  AMC  fleet  are  expected  to  have  the 
capability  to  transport  outsized  cargo  and  military  vehicles  in  addition  to  palletized  cargo.  For 
instance,  the  C-5  is  capable  of  carrying  two  Abrams  main  battle  tanks,  an  Abrams  tank  plus 
two  Bradley  armored  fighting  vehicles,  10  LAV  light  armored  vehicles,  six  Apache  attack 
helicopters,  or  36  standard  pallets,  type  463L  (Bolkcom,  2007).  The  volumetric  load  factor 
limitation  for  the  new  aircraft  based  on  AMC  operations  listed  in  the  GATES  dataset  is  a 
simple  and  indirect  way  of  ensuring  that  the  new  aircraft  design  meets  outsized  cargo 
requirements. 

Method  and  Approach 

The  consideration  of  the  simultaneous  design  of  a  yet-to-be-introduced  aircraft  and 
operations  of  the  new  aircraft,  presents  significant  computational  challenges.  We  adapt  a 
previously  used  decomposition  strategy,  with  aviation  applications  including  commercial 
airlines,  fractional  management  companies,  and  air  taxi  services  (Mane  &  Crossley,  2006, 
2012;  Mane  etal.,  2007). 

Subspace  Decomposition  Approach 

The  decomposition  strategy,  as  shown  in  Figure  4,  decomposes  the  MINLP  problem 
into  smaller  optimization  problems — each  sub  problem  follows  the  natural  boundaries  of 
disciplines  involved  in  formulating  the  original  problem.  Prior  research  (Mane  et  al. ,  2007) 
has  applied  this  decomposition  approach  to  the  case  of  a  commercial  air  transportation 
problem  where  the  objective  is  to  design  a  yet-to-be-introduced  aircraft  that  minimizes  fleet- 
level  operating  cost  while  meeting  passenger  demand  travel  obligations.  Here,  we  adapt  the 
same  decomposition  approach,  adapted  to  the  AMC  airlift  scenario.  The  top-level  problem, 
shown  in  Figure  4,  coordinates  the  aircraft  sizing  and  fleet  allocation  subproblems. 
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Top  Level 

minimize:  Fleet  DOC 


Ranges 
Pallet x 


Aircraft  Sizing  Subspace 
minimize:  DOC  on  route  (Range*) 

subject  to:  takeoff  distance 

variables:  ARX,  ( 77  If)*,  ( IF/S)* 


variable:  Pallety  Range x 


Pallet x 


AMC  Fleet  Allocation  Subspace 


minimize:  fleet  DOC 


Fleet  DOC 


subject  to:  pallet  capacity. 


trip  limits 
variables:  xxi 


Figure  4.  Subspace  Decomposition  of  MINLP  Problem 


Top-Level  Optimization 

The  top-level  problem  seeks  to  minimize  the  fleet-level  DOC  using  pallet  capacity  (an 
integer)  and  design  range  (continuous)  of  the  new,  yet-to-be-introduced  aircraft  type  X  as 
the  decision  variables;  the  optimization  problem,  at  this  stage,  is  addressed  using  a  simple 
enumeration  scheme.  A  quasi-enumeration  approach  of  varying  pallet  capacity  in 
increments  of  one  and  design  range  in  increments  of  200  nmi  reduces  computational  time, 
albeit  with  the  possibility  of  reduced  resolution  of  the  design  space.  However,  the  quasi¬ 
enumeration  approach  maps  out  correct  trends  for  the  objective  function  topology  in  the 
solution  space.  Heuristic  algorithms  such  as  Simulated  Annealing  (SA),  Genetic  Algorithms 
(GA),  and  so  forth,  may  be  needed  to  solve  the  small  MINLP  top-level  optimization  problem 
for  studies  involving  more  computationally  intensive  and  larger  sized  top-level  problem 
formulations.  These  top-level  decision  variables  are  essentially  “design  requirements”  for 
the  new  cargo  aircraft  design. 

Aircraft  Sizing  Subspace 

The  pallet  capacity  and  design  range  of  the  yet-to-be-introduced  aircraft  from  the  top- 
level  problem  then  become  inputs  to  the  aircraft  sizing  problem.  Here,  the  aircraft  sizing 
problem  seeks  to  minimize  the  direct  operating  cost  of  the  new  yet-to-be-introduced  aircraft, 
subject  to  constraints  on  minimum  take-off  distance.  Operating  cost  is  the  aircraft  objective 
here  because  it  matches  the  top-level  objective  for  minimum  fleet  cost. 

The  aircraft  design  variables  are  aspect  ratio(^)v,  thrust-to-weight  ratio  (T /Wj 

and  wing  loading  (W / S)  There  are  many  other  design  variables,  but  these  three  have 

significant  impact  on  the  size,  weight,  and  performance  of  the  aircraft.  The  objective 
function  can  be  altered  to  minimize  alternative  objectives  such  as  fuel  burn,  and  be  subject 
to  additional  constraints  as  required.  The  aircraft  sizing  problem  is  a  nonlinear  programming 
problem  (NLP)  and  described  by  Equations  15-20. 


Minimize  /-(ZXX7ra@B)x 


(15) 


Subject  to 


ST0 (Pallet x, (A R) x,(W/S)x,(T/W)x)<D  (aircraft  take-off  distance)  (1 6) 
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6  <  Pallet x  <  36 

(17) 

6.0<(z/tf)v<9.5 

(18) 

65<(W/  S)  v<\6\ 

(19) 

0.18<(7/fF)v  <0.35 

(20) 

After  finding  the  aircraft  that  leads  to  the  lowest  operating  cost  for  the  aircraft  design 
range,  the  aircraft  performance  is  predicted  for  the  routes  in  the  cargo  network.  The 
allocation  subproblem  then  uses  the  cost  coefficients  for  the  new  aircraft,  Cxi,  Cxi,  Cxi, 
together  with  the  top-level  design  variables,  design  range,  and  pallet  capacity,  as  inputs. 

Determination  of  Number  of  New  Aircraft 

The  number  of  new  aircraft  to  be  introduced  to  the  existing  fleet  is  unknown  a  priori, 
because  the  capacity  of  the  new  aircraft  is  described  by  the  top-level  design  variable, 

Palletx.  However,  the  AMC  strategic  fleet  is  expected  to  be  capable  of  servicing  the 
maximum  possible  demand  scenario  by  requirement.  AMC  force  structure  programmers  use 
the  metric  million-ton-miles  per  day  (MTM/D)  when  funding  out-year  aircraft  purchases,  and 
many  civilian  agencies  are  accustomed  to  visualizing  fleet  capability  in  terms  of  MTM/D  (Air 
Mobility  Command,  2010).  The  Mobility  Capabilities  and  Requirement  Study  (MCRS)  2016 
(Jackson,  2009)  illustrates  three  different  scenarios  that  the  capacity  of  the  strategic  fleet 
must  always  meet.  The  peak  for  MCRS  Case  1 ,  which  represents  the  highest  level  of 
modeled  strategic  airlift  demand,  required  32.7  MTM/D.  MTM/D  values  for  each  type  of 
aircraft  are  calculated  using  empirical  data.  A  C-5  carries  0.1209  MTM/D.  The  newer  C-17 
carries  0.1245  MTM/D  (Kopp,  2004).  The  Boeing  747-F  carries  0.1705,  but  is  not  included  in 
calculating  strategic  airlift  fleet  MTM/D,  because  AMC  does  not  operate  the  Civil  Reserve  Air 
Fleet  (CRAF).  Hence,  the  747-F  does  not  affect  the  number  of  new  aircraft  X  required  to 
meet  the  peak  demand.  MTM/D  of  the  new  aircraft  X  is  calculated  using  Equation  21 .  The 
resulting  value  is  then  used  to  compute  the  number  of  new  aircraft  X  required. 


MTM/D  = 


Block  speed  x  Average  payload  x  UTE  rate  x  Productivity  Factor 

1,000,000 


(21) 


The  utilization  rate  (UTE  rate)  of  the  new  aircraft  is  assumed  to  be  12  hr/day,  and  a 
productivity  factor  of  4.8  is  assumed  for  the  new  aircraft,  which  is  within  the  typical  range  of 
the  strategic  airlift  fleet  average  value. 

AMC  Fleet  Allocation  Subspace 

The  cost  of  operating  the  yet-to-be-introduced  aircraft  type  X  on  individual  routes, 

Cxi,  and  with  pallet  capacity  Palletx  are  constants  in  the  aircraft  allocation  problem.  Here,  the 
objective  is  to  minimize  the  fleet-level  direct  operating  costs  using  characteristics  of  the 
existing  and  yet-to-be-introduced  aircraft  (cost  coefficients  for  each  route,  pallet  capacity). 
Constraints  are  set  such  that  the  number  of  trips  per  aircraft  does  not  exceed  the  trip  limit  for 
each  aircraft  type,  and  the  combined  capacity  of  all  aircraft  provided  meets  the  demand  on 
each  route.  The  allocation  subproblem  equations  are  described  by  Equations  22-25.  As 
described  previously,  this  approach  assumes  an  aircraft  round  trip  assumption,  which 
removes  the  need  for  a  node  balance  constraint;  this  means  the  capacity  enforced  by 
Equation  24  will  be  sufficient  to  carry  the  largest  demand  between  the  two  bases  connected 

by  route  i.  The  local  decision  variables  in  the  allocation  problem,  XA — the  numbers  of  trips 
made  by  aircraft  typeX  on  route  i — are  integers,  making  the  allocation  problem  an  integer 
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programming  (IP)  problem.  The  Generic  Algebraic  Modeling  System  (GAMS)  software 
package,  accessed  through  a  MATLAB  interface,  was  used  to  solve  the  allocation  problem, 
using  the  CPLEX  solver  option  (Ferris,  1998.) 


Minimize 


Fleet  DOC  = 


Z 


Z  [CMXAi]  ’ 


1=1  AeC-5, 

C- 17, 747-F, X 


(22) 


Subject  to  Z ^  =  C-5, C-l 7, 747-F,  X  (trip  limits/aircraft  count)  (23) 

i=l 

Z  CaPnxAi  -  Q  (capacity)  (24) 

A=C-5,  C- 17, 

B-747,X 

X Ai  e  int  ,  >  0  (25) 

Solution  for  Cases 
Example  Problem  Solutions 

The  MDO  decomposition  method  reduces  the  computational  cost  by  separating 
discipline-specific  analysis  of  problems.  As  described  previously,  the  route  network  for  the 
three-route  example  uses  data  from  GATES  for  demand  and  to  set  trip  limits.  The  objective 
was  to  minimize  fleet  DOC  for  a  representative  year  of  operating  the  fleet.  The  actual  size  of 
the  strategic  airlift  fleet  dedicated  to  cargo  transport  was  obtained  from  GATES  dataset  by 
identification  of  unique  tail  numbers,  resulting  in  fleet  composition  of  92  C-5s,  145  C-17s, 
and  69  747-Fs.  Because  this  three-route  problem  is  much  smaller  than  the  full  network 
reconstructed  from  the  GATES  dataset,  the  number  of  aircraft  and  the  fleet-level  MTM/D 
value  for  the  three-route  problem  were  reduced  proportionally  to  the  pallet  demand  from  the 
entire  GATES  dataset  pallet  demand.  The  reduced  fleet  consists  of  four  C-5  aircraft,  five  C- 
17  aircraft,  and  three  747-Fs.  Each  aircraft  type  is  limited  to  a  trip  limit  value  calculated  from 
the  GATES  dataset  by  extracting  the  number  of  trips  made  by  each  type  of  aircraft  per  year. 
The  C-17  has  a  limit  of  53  trips  per  year  per  aircraft,  the  C-17  has  a  limit  of  103  trips  per 
year  per  aircraft,  and  the  747-F  is  limited  to  69  trips  per  year  per  aircraft.  Because  the 
utilization  rate  of  an  aircraft  depends  highly  on  the  aircraft’s  age,  the  newly  designed 
aircraft’s  trip  limit  is  assumed  to  be  1 1 0%  of  the  highest  trip  limit  in  the  existing  fleet,  or  1 1 3 
trips  per  year  per  aircraft.  These  trip  limits  ensure  that  the  allocation  does  not  exceed  the 
number  of  available  aircraft.  Figure  5  shows  the  results  of  the  partial  enumeration  employed 
for  the  top-level  problem,  and  Table  3  summarizes  the  solution  obtained  for  the  example 
three-route  network. 
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Figure  5.  DOC  Variation  for  the  Top-Level  Design  Space  for  the  Three-Route  Problem 


Table  3.  Solution  for  the  Example  Problem 


Variable,  Constraint, 
Objective 

Baseline  Allocation 

Allocation  &  Design 
Solution 

Xc-5,i  (trips  by  C-5s  on  Route  1) 

126 

0 

xC-5,2  (trips  by  C-5s  on  Route  2) 

0 

167 

Xc-5,3  (trips  by  C-5s  on  Route  3) 

86 

0 

Xc-i7i  (trips  by  C-17s  on  Route 

1) 

1 

0 

Xc-i7  2  (trips  by  C-17s  on  Route 

2) 

236 

1 

Xc-i7  3  (trips  by  C-17s  on  Route 

3) 

1 

1 

x747_f,i  (trips  by  Boeing  747-F  on 
Route  1) 

0 

0 

X747-f,2  (trips  by  Boeing  747-F  on 
Route  2) 

117 

0 

X747-F.3  (trips  by  Boeing  747-F  on 
Route  3) 

90 

0 

xXi  (trips  by  aircraft  X  on  Route 

1) 

133 
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xX2  (trips  by  aircraft  X  on  Route 
2) 

49 

xX3  (trips  by  aircraft  X  on  Route 

3) 

157 

Number  of  new  aircraft  X 
introduced 

3 

Rangex,  nautical  miles 

- 

1,000 

Palletx 

- 

36 

(W/S)x,  lb/ft2 

- 

104.2 

(T/W)x 

- 

0.208 

ARX 

- 

6.00 

Total  pallet  capacity  on  Route  1 

4,554 

4,788 

Total  pallet  capacity  on  Route  2 

7,641 

7,794 

Total  pallet  capacity  on  Route  3 

5,724 

5,670 

Fleet  DOC  for  one  year 

$49,458,132 

$  28,304,998 

DOC  saving  from  baseline 

- 

42.77  % 

Fleet  fuel  cost  for  one  year 

$  21,716,142 

$  11,597,685 

Fuel  cost  saving  from  baseline 

- 

46.59  % 

The  baseline  scenario  describes  the  current  fleet  operation  without  the  introduction 
of  the  new  aircraft  type  X.  The  results  obtained  for  this  allocation  problem  provide  a  baseline 
to  measure  the  effectiveness  of  introduction  of  the  yet-to-be-designed  aircraft  in  the  fleet 
mix.  The  allocation  problem  from  the  baseline  scenario  results  in  a  $49,458,132  fleet  DOC 
per  year.  For  these  two  solutions,  the  fleet-level  fuel  consumption  is  also  available.  With 
the  newly  introduced  type  X  aircraft,  the  fleet  uses  almost  47%  less  fuel.  However,  this 
approach  clearly  customizes  the  new  aircraft  to  the  route  network  and  demand  structure.  As 
a  result,  the  new  aircraft  X  is  a  short-range  aircraft  with  a  very  large  volume;  this  enables 
fewer  flights  of  this  smaller  aircraft  to  meet  demand.  Figure  6  emphasizes  this  result  by 
including  the  new  aircraft’s  payload-range  performance  along  with  the  existing  aircraft. 
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Figure  6.  Payload  Range  Curves  for  Existing  Aircraft  and  Optimal  Aircraft  Sizing 

Solution  for  Three-Route  Problem 

AMC  Expanded  Network  Solution 

Symmetric  Demand  Network 

The  three-route  problem  provides  a  simplistic  example  of  AMC  operations  to 
illustrate  the  approach  and  demonstrate  the  ability  to  generate  solutions.  Increasing  the  size 
of  the  network  to  investigate  the  ability  to  solve  larger  and  more  complex  network  system 
problems  using  decomposition  is  appropriate.  Our  current  formulation  assumes  a  round  trip 
assumption,  where  each  aircraft  flies  from  an  origin  base  to  a  destination  base  and  then 
returns  to  the  same  origin;  this  is  a  reasonable  assumption  under  symmetric  demand 
conditions,  which  was  appropriate  for  previous  commercial  passenger  airline  work. 

However,  many  of  the  routes  in  the  AMC  network  do  not  have  symmetric  demand,  because 
most  cargos  are  transported  one  way.  To  study  the  effects  of  asymmetric  demand  and 
effectively  address  this  issue,  we  developed  a  metric  that  calculates  the  asymmetry  between 
origin  destination  pairs  (O-D  pairs). 

\Demand n  n  -  Demand n  n\ 

Demand  asymmetry  =  — ' - — - xlOO  (26) 

ma x(Demand 0  D, Demand D  0) 

This  approach  would  be  zero  if  the  demand  was  symmetric.  With  demand 
asymmetry  calculated  on  each  route,  the  routes  with  a  demand  asymmetry  greater  than 
25%  are  filtered  from  the  route  network  before  implementing  the  decomposition  approach  to 
simultaneously  design  the  new  cargo  aircraft  while  also  allocating  the  fleet  to  meet  demand. 
Of  the  701  routes  in  the  full  network  reported  in  GATES,  1 1 1  routes  have  a  demand 
asymmetry  of  less  than  25%.  This  set  of  filtered  routes  represents  16%  of  total  routes  and 
28%  of  the  pallets,  and  has  an  average  of  11%  demand  asymmetry 

As  the  size  of  the  route  network  and  demand  increased  from  the  three-route 
problem,  the  numbers  of  aircraft  available  for  use  in  the  problem  also  increased  in 
proportion  to  the  demand  increase.  The  existing  fleet  in  this  symmetric  demand  problem 
comprises  27  C-5s,  42  C-17s,  and  20  747-Fs.  The  MTM/D  value  is  also  increased  in 
proportion  to  demand  decrease  to  have  more  aircraft  type  X  introduced  to  the  fleet.  Table  4 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-  489- 


summarizes  the  solution  obtained  for  the  symmetric  demand  route  network  (although 
without  the  per-route  detail,  given  the  size  of  the  problem),  and  Figure  7  presents  the  partial 
enumeration  scheme  to  solve  the  top-level  problem. 


Table  4.  Solution  for  the  Symmetric  Demand  Problem 


Variable,  Constraint, 
Objective 

Baseline  Allocation 

Allocation  &  Design 
Solution 

xC-5  (trips  by  C-5) 

1,431 

1,431 

xc-i7  (triPs  by c_17) 

3,074 

344 

x747_F  (trips  by  747-F) 

1,378 

1,380 

xx  (trips  by  aircraft  X) 

- 

1,469 

Number  of  aircraft  X 
introduced 

13 

Rangex ,  nautical  miles 

- 

2,200 

Palletx 

- 

35 

(W/S)x,  lb/ft" 

- 

113.6 

(T/W)x 

- 

0.227 

> 

73 

X 

- 

6.15 

Fleet  DOC  for  one  year 

$595,393,013 

$469,500,435 

DOC  saving  from  baseline 

- 

21.14  % 

Fleet  fuel  cost  for  one  year 

$297,067,262 

$231,347,251 

Fuel  cost  saving  from 
baseline 

22.12  % 
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Figure  7.  DOC  Variation  for  the  Top-Level  Design  Space  for  the  Symmetric  Demand 

Problem 

With  the  dataset  at  hand,  the  allocation  of  the  problem  with  the  introduction  of  aircraft 
type  X  was  investigated.  The  resulting  optimal  design  variable  at  the  top  level  suggests  an 
aircraft  design  capacity  of  35  pallets  and  a  design  range  of  2,200  nautical  miles.  The  aircraft 
sizing  subproblem  result  suggests  aircraft  type  X  design  with  the  wing  loading  of  113.6 
Ib/ft2,  aspect  ratio  of  6.15,  and  thrust-to-weight  ratio  of  0.227.  The  allocation  subproblem 
introduces  13  aircraft  type  X  in  the  fleet  and  provides  DOC  savings  of  21.14%  and  fuel  cost 
savings  of  22.12%  compared  to  the  allocation  of  the  fleet  without  the  new  aircraft  for  this 
symmetric  demand  scenario.  These  results  also  indicate  a  comparatively  short-range 
aircraft  with  a  high  pallet  capacity.  As  apparent  from  Figure  8,  this  solution  also  requires 
some  of  the  existing  aircraft  to  perform  longer  range  routes,  while  the  fleet  cost  and  fuel 
savings  result  by  using  the  newer  aircraft  on  shorter  routes. 


Figure  8.  Payload-Range  Curves  for  Existing  Aircraft  and  Optimal  Aircraft  Sizing 

Solution  for  Symmetric  Demand  Problem 


m  khsiikm 
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Full  Network 

Having  presented  applicability  of  the  decomposition  strategy  to  the  symmetric 
demand  problem,  the  full  AMC  network  problem  was  attempted.  Routes  and  pallet  demand 
from  the  entire  GATES  dataset  were  considered  in  this  full  network  problem.  The  fully 
considered  AMC  service  network  has  a  66%  demand  asymmetry  (based  on  Equation 
1 1  ).Thus,  the  round  trip  assumption  may  not  be  reflective  of  actual  operations,  but  the 
constraints  will  ensure  there  is  sufficient  capacity  in  both  directions  on  a  route,  even  if  one 
direction  has  a  substantially  lower  demand.  With  this  potentially  limiting  assumption, 
addressing  this  problem  demonstrates  that  the  approach  can  scale  to  larger  problems,  in 
terms  of  routes  served.  In  the  full  network  problem,  the  round  trip  assumption  implies  every 
trip  has  symmetric  demand  resulting  in  a  total  of  209,787  pallets  delivered  between  701 
routes.  Table  5  summarizes  the  solution  obtained  for  the  full  network  problem,  and  Figure  9 
illustrates  the  partial  enumeration  to  find  the  top-level  variables. 


Table  5.  Solution  for  the  Full  Network  Problem 


Variable,  Constraint, 
Objective 

Baseline  Allocation 

Allocation  &  Design 
Solution 

xC-5  (trips  by  C-5) 

4,876 

4,876 

Xc-i7  (trips  by  C-17) 

6,320 

303 

x747_F  (trips  by  747-F) 

4,753 

2,112 

xx  (trips  by  aircraft  X) 

- 

5,537 

Number  of  aircraft  X  introduced 

- 

49 

Rangex,  nautical  miles 

- 

2,400 

Palletx 

- 

36 

(W/S)x,  Ib/ft2 

- 

114.4 

(T/W)x 

- 

0.228 

> 

73 

X 

- 

6.23 

Fleet  DOC  for  one  year 

$1,743,525,560 

$1,370,781,919 

DOC  saving  from  baseline 

- 

21.38  % 

Fleet  fuel  cost  for  one  year 

$888,509,686 

$693,047,455 

Fuel  cost  saving  from  baseline 

- 

22.00  % 
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«  10 


Range  (miles)  Pallet  Capacity 

Figure  9.  DOC  Variation  for  the  Top-Level  Design  Space  for  the  Full  Network  Problem 

The  results  suggest  the  introduction  of  49  aircraft  type  X  to  the  existing  fleet  with  a 
maximum  pallet  capacity  of  36,  using  the  design  pallet  weight  of  4,003  pounds  to  set  the 
volume  of  the  fuselage  and  design  range  at  MTOW  of  2,400  nautical  miles.  The  new  aircraft 
again  mainly  service  the  shorter  routes  in  the  route  network  as  evidenced  in  Figure  10.  The 
wing  loading  of  aircraft  X  is  1 14.4  lb/ft2,  the  aspect  ratio  is  6.23,  and  the  thrust-to-weight  ratio 
of  aircraft  X  is  0.228,  which  is  a  slight  increase  compared  to  the  solution  from  the  symmetric 
demand  scenario  due  to  a  slight  increase  in  fuselage  size  and  design  range. 


Figure  10.  Payload  Range  Curves  for  Existing  Aircraft  and  Optimal  Aircraft  Sizing 

Solution  for  Full  Network  Problem 


m  khsiikm 
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Future  Work  and  Conclusions 

The  studies  presented  here  assume  simplified  demand  scenarios  to  demonstrate  the 
viability  and  applicability  of  the  decomposition  approach  in  solving  problems  that  represent 
AMC  operations,  and  as  a  tool  to  better  inform  acquisition  decisions.  AMC  operations 
typically  involve  highly  uncertain  cargo  demand  operations — a  contrasting  difference  to 
airline  problems  that  are  fairly  constant.  The  uncertainties  in  cargo  demands  and  shipping 
priorities  manifest  as  uncertainties  in  the  load  factor  and  quantity  of  cargo  flow  between  O-D 
pairs. 

The  uncertainties  in  load  factor  and  total  cargo  can  be  modeled  using  a  Monte  Carlo 
sampling  technique.  This  model  addresses  the  uncertainty  in  both  demand  and  load  factor, 
within  a  probabilistic  framework.  Through  addressing  uncertainty  via  a  Monte  Carlo  sampling 
technique,  the  subspace  decomposition  method  can  determine  a  yet-to-be-introduced 
aircraft  design  that  is  tailored  to  minimize  fleet-level  cost  (fuel/direct  operating)  under 
prescribed  uncertainty.  Future  work  will  reflect  a  more  representative  mixture  of  the  AMC 
fleet  from  the  GATES  dataset  with  uncertainty  in  the  operational  characteristics  of  the  fleet 
and  route  network. 


Figure  11.  Subspace  Decomposition  of  MINLP  Problem  With  Uncertainty 

The  round  trip  assumption,  although  valid  for  studies  with  a  symmetric  demand  route 
network,  appears  to  be  a  weak  abstraction  for  the  entire  network,  as  mentioned  earlier. 
Future  work  will  consider  “scheduling-like”  formulations  for  the  resource  allocation  problem 
by  implementing  node  balance  constraints.  The  addition  of  node  balance  constraints  would 
increase  the  computational  complexity  and  possibly  the  computational  burden,  as  individual 
aircraft  tail  numbers  need  to  be  tracked  in  the  model.  However,  this  formulation  allows 
modeling  of  varying  directional  pallet  demand  between  origin  destination  pairs.  An 
acquisition  support  issue  is  the  selection  of  the  top-level  design  variables  that  represent 
some  of  the  requirements  for  a  new  platform.  Payload  capacity,  design  cruise  velocity,  and 
range  are  common  aircraft  design  variables  and  are  logical  choices  for  these  top-  or  system- 
level  variables.  Future  investigations  will  consider  other  platform  requirement  variables  as 
necessary.  The  resulting  values  for  these  requirement  variables  can  inform  acquisition 
decisions  about  what  new  platform  requirements  will  lead  to  a  more  successful  fleet. 
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The  studies  presented  here  also  use  direct  operating  cost  as  the  objective  function. 
This  follows  from  the  previous  work  for  commercial  airline-related  investigations,  but  here 
this  allows  for  the  chartered  747-F  aircraft  to  be  modeled  as  part  of  the  problem.  If  a 
formulation  sought  to  minimize  fuel  consumed  by  AMC,  it  is  possible  that  one  solution  would 
lead  to  carrying  all  cargo  on  the  chartered  747-F  aircraft.  As  demonstrated  previously,  fleet- 
level  fuel  values  are  readily  available  and  minimizing  DOC  has  a  strong  relationship  to 
minimizing  fuel  consumption. 

From  the  results,  all  of  the  newly  designed  aircraft  should  be  smaller  aircraft  than  the 
existing  aircraft  in  the  strategic  fleet.  This  diversifies  the  size  of  the  aircraft,  and  tries  to 
exploit  the  fact  that  existing  large-size  aircraft  generally  carry  only  a  small  fraction  of  their 
maximum  weight  (and  in  some  cases  volume)  capacity.  The  smaller  aircraft  will  be  used 
predominantly  on  routes  that  are  short  and  will  carry  a  comparatively  large  number  of  pallets 
per  flight.  In  comparison,  the  scenario  in  which  the  new  aircraft  design  and  allocation  relaxes 
the  load  factor  imposed  on  weight  suggests  an  even  smaller  aircraft  that  is  designed  to  carry 
only  a  small  number  of  palletized  cargos  weighing  approximately  4,000  lbs  each.  Results 
suggest  that  this  platform  will  be  even  more  efficient  as  many  of  the  routes  are  short  and 
day-to-day  base  cargo.  The  fuel  saving  in  all  cases  are  directly  related  to  the  DOC  saving  as 
fuel  cost  is  a  driving  factor  in  DOC. 

The  research  presented  in  this  paper  demonstrates  an  approach  to  concurrently 
design  a  yet-to-be-introduced  aircraft  and  its  fleet-level  operations  in  the  context  of  military 
airlift  operations.  The  decomposition  approach  presented  in  this  paper  makes  the  resulting 
MINLP  problem  tractable.  The  solution  space  of  the  top-level  optimization  problem  provides 
a  landscape  that  could  help  acquisition  practitioners  make  informed  acquisition  decisions 
and  design  choices  about  the  new  platform.  The  design  combination  of  the  top-level 
problem  corresponds  to  different  levels  of  fleet-level  direct  operating  costs,  and 
consequently,  different  operations  (allocation  of  fleet  over  service  network.) 

Although  the  studies  presented  here  focus  on  the  concurrent  design  of  aircraft  to 
improve  fleet-level  operational  performance  metrics,  the  problem  formulation  and  solution 
methodology  have  features  that  can  be  extended  to  other  systems  of  interest  and/or  the 
design  of  multiple  yet-to-be-designed  systems. 
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Abstract 

DoD  energy  inefficiency  is  a  significant  liability  and  a  constraint  on  operations  and  a  force- 
protection  challenge.  It  is  therefore  imperative  to  reduce  energy  demand  and  provide 
operational  forces  greater  flexibility  among  alternative  energy  sources.  However,  the  current 
acquisition  processes  undervalue  technologies  with  the  potential  to  improve  energy 
efficiency.  We  report  the  results  of  leveraging  an  innovative  platform,  the  Massive  Multiplayer 
Online  Wargame  Leveraging  the  Internet  (MMOWGLI)  to  link  and  elicit  collective  intelligence 
from  the  acquisition  community  for  the  challenge  of  DoD  energy  inefficiency.  We  first  linked 
the  existing  MMOWGLI  energy  data  with  samples  of  acquisition  data  using  lexical  link 
analysis  (LLA).  We  generated  match  matrices  based  on  themes  discovered  in  both  data  sets. 
The  themes  and  match  matrices  helped  identify  the  gaps  and  opportunities  to  apply  collective 
intelligence  from  the  MMOWGLI  game  to  the  current  acquisition  process.  This  effort 
demonstrates  superb  potential  of  an  innovative  methodology  that  can  be  deployed  quickly  to 
mobilize  the  intellectual  capacities  of  the  acquisition  community.  It  may  also  increase  the 
overall  awareness  of  ongoing  acquisition  research  to  warfighters  and  create  a  positive  impact 
for  the  future  acquisition  decisions  to  help  achieve  improved  DoD  energy  efficiency. 

Background,  Needs,  and  Research  Questions 

Studies  evaluating  the  DoD’s  energy  use  have  been  conducted  by  the  Institute  for 
Defense  Analyses,  the  Defense  Science  Board  Energy  Security  Task  Force,  and  JASON 
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(an  independent  scientific  advisory  group).  All  three  studies  suggest  that  DoD  energy 
inefficiency  is  a  significant  liability,  a  constraint  on  operations,  and  a  force-protection 
challenge.  More  specifically,  all  three  studies  led  to  two  consistent  requirements  for  DoD 
energy  efficiency:  (1)  By  reducing  energy  demand,  we  may  provide  operational  forces 
greater  flexibility  and  reduce  their  dependency  on  logistics  infrastructure;  and  (2)  We  can 
improve  the  DoD’s  current  requirements  and  acquisition  processes  to  value  the  technologies 
with  the  potential  to  improve  energy  efficiency  (DoD  Acquisition  and  Technology,  2012). 

The  Massive  Multiplayer  Online  Wargame  Leveraging  the  Internet  (MMOWGLI), 
sponsored  by  the  Office  of  Naval  Research  (ONR),  is  an  online  game  platform  designed  to 
elicit  collective  intelligence  from  an  engaged  pool  of  world-wide  players.  The  Naval 
Postgraduate  School  (NPS)  is  one  of  the  primary  developers  of  the  game  software. 

Recently,  the  Navy’s  Energy  and  Environmental  Readiness  Division  (OPNAV  N45),  hosted 
by  NPS  Modeling  Virtual  Environments  and  Simulation  (MOVES)  Institute,  conducted  a  civic 
and  military  collaboration  specifically  for  examining  Navy  energy  efficiency  May  22-25.  In 
the  past,  the  NPS  hosted  a  series  of  successful  games,  piracyMMOWGLI  (201 1-present, 
ongoing)  and  energyMMOWGLI  (May  2012),  which  built  the  critical  mass  of  players  needed 
to  find  creative  solutions  to  the  real-life  difficult  problems,  such  as  piracy  and  energy. 

In  the  energyMMOWGLI  game,  ideas  were  collected  through  “play  an  idea  card”  and 
“take  action,”  as  shown  in  Figure  1.  The  motivating  “call  to  action”  for  players  is  to  improve 
the  U.S.  Navy’s  combat  capability  and  energy  security,  particularly  by  promoting  energy 
efficiency,  reducing  energy  consumption,  and  diversifying  its  energy  supply  (use  of 
alternative  energy)  for  the  sake  of  future  strategic  readiness.  The  overall  goal  is  to  reduce 
reliance  on  fossil  fuels  from  overseas. 


Figure  1.  The  energyMMOWGLI  Game 


In  this  energyMMOWGLI  game,  560  players  contributed  over  5000  ideas  and  68 
action  plans.  Lexical  link  analysis  (LLA;  Zhao,  Gallup,  &  MacKinnon,  2010,  2011a,  2011b, 
2011c,  2012)  was  used  in  analyzing  the  collected  data.  All  results  are  published  online  (see 
MMOWGLI  Energy  Game,  2012;  MMOWGLI  Energy  Game  Portal,  2012;  MMOWGLI 
Business  Initiative  [Bll]  Game,  2013;  MMOWGLI  Bll  Game  Portal,  2013). 

•  https://portal.mmowgli.nps.edu 

•  https://portal.mmowgli.nps.edu/energy-welcome 

•  http://web.mmowgli.nps.edu/energy/ldeaCardChainEnergy2012.html 

•  http://web.mmowgli.nps.edu/energy/ActionPlanListEnergy2012.html 

We  leveraged  the  energyMMOWGLI  game  in  the  acquisition  community  through  the 
following  four-step  process.  Further  details  appear  later  in  this  paper  and  in  the  online  game 
portal. 


mt  sciivrio, 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-498- 


1 .  Prepare  acquisition  data.  Collate  key  terms  and  goal  statements  of  current 
acquisition  programs  within  the  congressional  budget  processes  for  use  by 
the  LLA  methodology. 

2.  Perform  link  analysis  and  correlation.  Compare  the  already-collected 
energyMMOWGLI  results  to  determine  action  plan  relevance  on  a  program- 
by-program  basis. 

3.  Design  new  capabilities  for  information  collection.  Define  questions  for  a 
continuation  round  of  the  energyMMOWGLI  game,  to  support  programmatic 
life-cycle  needs  of  the  acquisition  community. 

4.  Plan/conduct  follow-on  games.  Conduct  a  follow-on  game  focused  on  shared 
needs  of  many  energy  programs,  demonstrating  the  value  of  this  approach  in 
a  formal,  repeatable  way. 

Methodology 
MMOWGLI  Game 

The  game  is  built  using  a  unique,  open  source,  software  adaptation  of  the  Institute 
for  the  Future  (IFTF) — designed  game  to  simulate  a  real  world  “brainstorm.”  A  player  needs 
to  register  with  a  required  game  identification  (ID)  and  e-mail.  First  and  last  name  and  other 
personal  identification  information  (Pll)  are  not  required. 

The  game  starts  with  the  explanation  of  the  situation  and  allows  a  player  to  “play  an 
idea”  or  “take  action.”  Users  can  then  choose  to  input  an  idea  or  participate  in  the  discussion 
of  an  existing  idea  in  the  categories  of  “Innovate”  and  “Defend.”  The  discussion  can  be  in 
one  of  the  following  categories:  expand — build  on  this  idea  to  amply  the  impact;  counter — 
challenge  this  idea;  adapt — take  this  idea  in  a  different  direction;  explore — something 
missing?;  or  ask  a  question,  as  shown  in  Figure  2. 

In  the  end,  the  system  will  gather  collective  intelligence  that  resides  in  color-coded, 
tree-structured  sets  of  ideas  and  discussions  in  text  format  as  shown  in  Figure  3.  If  an  idea 
and  its  associated  discussion  have  merit,  which  is  determined  in  the  combination  of  the 
player’s  score  and  the  Game  Master’s  recommendation,  it  will  be  taken  into  a  separate  “take 
action”  board  for  further  planning  and  deliberation. 


Figure  2.  Categories  of  Ideas  Based  on  the  Styles  of  Responses 
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Figure  3.  Ideas  Collected  in  the  Color-Coded  Tree-Structured  Categories 

The  MMOWGLI  platform  is  suitable  for  tackling  a  broad  range  of  challenges  for 
national  security,  multiple  stakeholders,  and  challenges  for  small  or  big  communities  (e.g., 
corporations  and  research  communities  like  the  acquisition  system  community).  It  is  a 
configurable  innovation  platform  that  can  be  adapted  to  any  scenario.  For  example,  an 
aerospace  and  defense  company,  Raytheon,  is  considering  the  game  engine  for  use  within 
a  company  as  a  corporate  innovation  platform. 

Lexical  Link  Analysis 

LLA  is  a  form  of  text  mining  in  which  word  meanings  represented  in  lexical  terms 
(e.g.,  word  pairs)  can  be  represented  as  if  they  are  in  a  community  of  a  word  network  (Zhao 
et  al. ,  2010,  2011a,  2011b,  2011c,  2012).  LLA  “discovers”  and  displays  these  networks  of 
word  pairs  from  large-scale  unstructured  data.  It  can  be  installed  as  a  search  and 
knowledge  management  tool  for  scoring  and  ranking  interesting  information  and  for 
visualizing  and  reporting  correlations  among  categories  and  layers  of  information  including 
lexical,  semantic,  and  social  links.  This  effort  then  presents  the  decision-maker  with 
previously  unavailable  and  emerging  patterns  and  themes,  as  well  as  unprecedented  levels 
of  analysis,  thus  reducing  the  workload  and  overcoming  the  blind  spots  of  human  analysts 
and  with  potential  automation.  For  example,  for  the  recent  MMOWGLI  games  used  to 
develop  and  identify  new  ideas  about  stated  subject  matters,  LLA  was  leveraged  to  identify 
potentially  interesting  information  from  “idea  cards,”  link  them,  then  recommend  them  to  the 
matched  action  plans  for  Game  Masters. 

Figure  4  shows  the  game’s  content  and  attributes,  which  were  processed  into  the 
inputs  (i.e. ,  meta_data.txt  and  a  directory  of  text  files  with  idea  card  contents  to  LLA). 
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Figure  4.  Idea  Cards  Transformed  to  LLA  Inputs  (e.g.,  a  Directory  With  Files  of  Content 

of  the  Cards  and  Attributes,  meta_data.txt) 

There  are  two  steps  used  in  LLA  to  discover  themes.  A  theme  is  a  cluster  of  related 
word  pairs: 

•  1st  Iteration  (Figure  5  (a)):  Compute  word  pair  clusters  using  Newman 
community  finding  algorithm — words  as  in  a  community  (Girvan  &  Newman, 
2002). 

•  2nd  Iteration  (Figure  5  (b)):  Select  lexical  terms  linked  to  the  most  central 
nodes,  for  example,  “fuel,  shipboard,  liquid.” 
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Figure  5.  Two  Steps  LLA  Iterations  to  Group  Word  Pairs  Into  Themes 


Research  Results 

As  shown  in  Figure  6,  in  Phase  I,  we  planned  to  demonstrate  the  feasibility  of  the 
social  media  energyMMOWGLI  game  as  an  innovation  platform  that  could  generate 
valuable  and  unexpected  contributions  and  solutions  towards  the  DoD  energy  efficiency 
through  the  acquisition  process  by  linking  the  current  acquisition  programs  with  the 
energyMMOWGLI  game  using  LLA.  We  achieved  this  objective  through  performing  the 
tasks. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  C FLANGE 


-  502- 


Figure  6.  A  Glance  of  the  Proposal  Objective 


Task  1:  Prepare  Acquisition  Data 

The  goal  here  is  to  collate  key  terms  from  the  current  acquisition  program  in  the 
congressional  budget  process.  The  congressional  budget  process  documents  (e.g.,  program 
elements  [PEs]  from  http://www.dtic.mil/descriptivesum)  will  be  used  in  this  task.  This  source 
is  the  accurate  and  authoritative  high-level  artifacts  under  the  DoD  Research,  Development, 
Test,  and  Evaluation  (RDT&E).  We  had  analyzed  part  of  these  documents  in  the  past  (Zhao 
et  al.,  2010,  2011a,  2011c,  2012)  in  detail  using  the  LLA  method  jointly  with  other  measures 
such  as  cost,  schedule,  and  performance. 

Specifically,  we  collected  the  following  most  recent  (2013)  tri-service  PE  documents 
for  this  project: 

•  http://www.dtic.mil/descriptivesum/Y201 3_Navy.html 

•  http://www.dtic.mil/descriptivesum/Y201 3_AirForce.html 

•  http://www.dtic.mil/descriptivesum/Y201 3_Army.html 

Task  2:  Perform  Analysis  and  Correlation 

Compare  the  already  collected  energyMMOWGLI  results  to  determine  action  plan 
relevance  on  a  program-by-program  basis. 

We  linked  the  energyMMOWGLI  data,  specifically,  38  action  plans  with  the  PEs 
prepared  in  Task  1 ,  and  224  Navy  PEs  to  evaluate  the  current  Navy  programs  relevant  to 
the  game  data.  Figure  7  shows  that  the  process  resulted  in  a  relevance  and  correlation 
matrix  as  illustrated. 
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Phase  I  Project:  Program  Relevance/Correlation  Matrix  of  MMOWGLI  Game 
Plans- Comparison  of  Key  Terms  from  Both  Sources 
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Figure  7.  Phase  I  Relevance  Matrix 


Figure  8.  The  Overall  Match  Matrix  for  the  MMOWGLI  Energy  Game  Action  Plans  and 

Navy  2013  Program  Elements 

Figure  8  shows  sorted  Navy  PEs  that  match  the  MMOWGLI  game  data  based  on  a 
sorted  LLA  score.  The  top  five  most  relevant  PEs  are  listed  as  follows: 

•  PE  0603724N:  Navy  Energy  Program 

•  PE  0601 153N:  Defense  Research  Sciences 

•  PE  06021 23N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Combat  Services  Support 
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Clicking  on  the  online  link  for  the  top  one  leads  to  the  online  page  of  the  “Navy 
Energy  Program,”  which  is  an  overall  PE  specifically  focusing  on  Navy  energy  issues  as 
shown  in  Figure  9.  This  validates  that  the  LLA  extracted  the  relevant  keywords  from  the 
game  data. 


The  matrix  in  Figure  8  shows  a  holistic  picture  of  the  current  acquisition  programs  in 
connection  with  the  DoD  energy  inefficiency  situations,  efficiency  requirements,  and 
possible  innovative  solutions.  Directly  looking  into  the  match  matrix,  as  illustrated  in  Figure 
8,  can  be  overwhelming.  For  that,  we  applied  LLA  to  discover  the  themes  and  divide  a  single 
match  matrix  into  many  match  matrices  in  different  themes.  For  our  research,  a  theme  is  a 
network  or  community  of  word  pairs  that  are  related  to  each  other.  To  discover  themes,  we 
first  applied  LLA  to  compute  word  pair  clusters  using  Newman  community  finding 
algorithm — words  as  in  a  community  (Girvan  &  Newman,  2002).  There  we  select  lexical 
terms  linked  to  the  most  central  nodes.  For  example,  shown  in  Figure  11,  the  red  nodes  are 
the  most  central  nodes  “environmental,  ship,  and  effective.”  The  red  links  are  the  word  pairs 
shared  by  both  sources  PEs  and  MMOWGLI  game  action  plans;  the  yellow  links  are  the 
word  pairs  unique  to  the  game  data;  and  the  green  ones  are  those  unique  to  the  PEs. 
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Figure  10.  Themes  Discovered  for  Navy  2013  Program  Elements  Documents  and 
energyMMOWGLI  Data,  Thresholded  and  Then  Sorted  According  to  the 
Overlapped  Word  Pairs  From  the  Two  Sources 
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Red  nodes:  the  most  "central"  nodes,  used  as  keywords  to  summarize  this  theme 
Red  Links:  shared  by  at  least  two  sources 

Green  Links:  from  Source  ARPM_mmowgli_energy 
Blue  links:  from  Source 
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Figure  11.  Theme  395(E)  Link-Strength  Visualizations:  “Environmental,  Ship,  &  Effective” 

A  separate  matrix  can  be  constructed  for  each  theme  for  the  word  pairs  that  belongs 
to  only  a  theme.  Figure  12,  the  correlation  matrix  for  Theme  395(E)  labeled  as 
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“environmental,  ship,  &  effective,”  which  has  the  highest  matched  word  pairs  in  Figure  12. 
The  matched  PEs  are  sorted  according  to  the  number  of  matched  action  plans.  For 
example,  the  top  matched  PE  is  “0603724N_PB_2013,”  titled  “Navy  Energy  Program,” 
indicating  that  there  is  a  current  Navy  program  dedicated  to  “energy.” 

We  used  this  matrix  to  determine  where  opportunities  reside  in  the  current  process  to 
include  energy-related  elements.  Also  shown  in  Figure  12,  two  concepts,  “energy  efficient” 
and  “ship  design,”  are  dominant  in  this  theme.  They  are  dominant  because  there  are  four  (4) 
and  two  (2)  out  of  38  action  plans  contain  word  pairs  “energy  efficient”  and  “ship  design,” 
respectively.  This  seems  to  suggest  that  “efficient  energy”  may  have  to  work  with  the 
concept  “ship  design.”  Plowever,  among  the  12  PEs  that  mentions  “ship  design,”  only  one 
entry  mentions  “energy  efficient.”  This  indicates  that  there  is  a  gap,  or  a  DoD  energy 
inefficiency  area,  and  therefore  an  opportunity  to  emphasize  the  concept  “energy  efficient”  in 
all  the  PEs  related  to  the  concept  “ship  design.” 
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Figure  12.  Match  Matrix  for  Theme  395(E) 


Following  the  same  analysis,  Appendix  A  lists  more  gap  and  opportunity  areas 
discovered  by  LLA. 

In  the  near  future,  we  will  engage  the  students,  faculties,  and  a  wide  acquisition 
research  community  to  continue  the  discussion  of  the  DoD  energy  efficiency  and  possible 
solutions  through  series  of  planned  MMOWGLI  games  (MMOWGLI  Energy  Game  Portal, 
2012).  As  possible  acquisition  professionals  being  Game  Masters,  the  brainstorming  and 
discussions  can  be  steered  towards  more  specific  requirements,  for  example,  the  ones 
below: 


1 .  Flow  to  provide  operational  forces  greater  flexibility  and  reduce  their 
dependency  on  logistics  infrastructure. 

2.  Flow  to  change  the  DoD’s  current  requirements  and  acquisition  processes  so 
they  do  not  undervalue  technologies  with  the  potential  to  improve  energy 
efficiency. 

The  results  from  the  match  matrices  can  be  recommended  areas  for  the  seed 
questions  for  a  MMOWGLI  energy  game. 

Conclusions 

Multiple  useful  conclusions  of  broad  applicability  arise  from  this  work. 
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•  We  demonstrated  the  use  of  the  MMOWGLI  social  media  brainstorming 
platform  and  LLA  as  a  combined  collective  intelligence  platform  to  gather 
consensus  via  the  MMOWGLI  energy  game  and  match  data  using  LLA,  with 
the  current  existing  DoD  programs,  derived  from  Navy  2013  PEs  documents. 

•  We  identified  critical  variables,  elements,  concepts,  or  word  pairs  that  can  be 
linked  to  Navy  energy  efficiency  within  and  among  numerous  programs. 

•  We  used  match  matrices  for  each  individual  theme  found  through  LLA  to 
identify  energy-related  parameters  or  elements  as  word  pairs,  and  then  we 
used  these  word  pairs  to  further  identify  opportunities  in  the  current  process, 
(i.e.,  what  PEs  might  be  good  candidates  to  engage  the  energy-related  action 
plans  discussed  in  the  MMOWGLI  energy  game?). 

•  We  found  that  the  great  majority  of  Navy  programs  are  affected  by  (or  even 
critically  dependent  on)  energy  issues,  but  goals  and  even  terms  are  handled 
inconsistently. 

Therefore,  without  imposing  significant  operational  burdens  and  vulnerabilities, 
innovative  “energy  efficiency”  ideas  from  the  social  media  game  might  be  quickly  and 
naturally  implemented  into  the  current  processes  that  drive  force  structures,  combat 
operations,  logistics,  and  acquisition  decisions. 

The  resulting  capability,  the  automation  of  LLA  computations  and  an  analyst 
interface  for  report  generation,  demonstrate  MMOWGLI  together  with  LLA  as  an  important 
tool  throughout  the  longer  life  cycle  of  the  acquisition  process  for  incorporating  the  “fully 
burdened  cost  of  fuel”  into  acquisition  analyses. 

Recommendations  for  Future  Work 

Much  work  can  continue;  specifically,  we  see  excellent  potential  in  the  following: 

•  Crowd  sourcing  to  provide  meaningful  feedback  on  either  cross-cutting 
themes  (such  as  energy  reduction/efficiency)  or  specific  acquisition 
programs. 

o  For  example,  acquisition  experts  might  participate  in  the  Business 
Innovation  Initiative  (bii)  MMOWGLI  Game  Round  2  in  summer  2013 
to  gain  further  experience  in  relevant  crowd-sourcing  capabilities. 

•  Building  MMOWGLI  game  infrastructure  in  tandem  with  LLA  computational 
structure  to  reduce  manual  labor  and  maximize  analyst  flexibility  with  each 
round. 

•  Continuing  work  on  real  datasets  that  spurs  meaningful  (rather  than  toy  or 
contrived)  analysis,  and  producing  further  data  visualizations  tuned  to  support 
focused  analytic  queries  by  players  and  decision-makers. 

•  Maintaining  backwards  compatibility  among  games  to  enable  steady  growth 
via  the  available  corpus  and  products  each  year.  This  further  enables 
longitudinal  analysis  and  observability  of  trends  and  evolution  over  time. 

•  Stabilizing  the  data-model  design  of  LLA  computational  products,  which  may 
enable  future  visualization  improvements  to  be  directly  applied  to  past 
products. 

•  Speedier  production  of  LLA  products  that  can  influence  fast-react  game 
rounds  or  program  changes  as  they  proceed,  rather  than  after  the  event.  We 
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want  to  reduce  analysis  cycles  from  weeks  to  days,  and  even  to  hours, 
approaching  real  time. 

•  Program-support  brainstorming  and  collective  intelligence  experiments  that 
should  continue,  both  for  proposed  and  current  programs  of  record.  Games  + 
link  analysis,  connecting  the  record  of  “what  is  reported  being  done”  with 
“what  do  people  think,”  all  help  normalize  the  use  of  concept  terminology  and 
also  identify  unsuspected  applicability  of  new  breakthrough  capabilities. 

•  Overall  progress  and  process  improvements  that  may  now  be  measured  so 
that  causes  and  effects  of  improvements  in  acquisition  system  cost- 
effectiveness  and  responsiveness  are  documented. 

•  Navy  strategies  for  improving  energy  efficiency  needs  to  be  handled 
consistently  across  programs.  Terms  of  reference,  metrics,  and  opportunities 
all  need  to  be  addressed  consciously  and  consistently. 

•  Following  a  series  of  deliberate  experiments,  long-term  procedural 
improvements  to  the  formal  milestone  acquisition  process  can  be  considered. 
For  example: 

o  Are  program  terms  of  reference  consistent  with  DoD-wide  best 
practice? 

o  Are  all  applicable  energy  reduction  and  energy  efficiency  techniques 
identified? 

o  Routine  crowd  sourcing  as  due  diligence:  subject-matter  expert  and 
public  reviews  (as  appropriate)  to  accompany  milestone  decisions. 

o  Has  in-game  or  post-game  analysis  identified  synergies  among 
different  programs  that  deserve  further  investigation? 

•  Open  question:  How  can  these  tools  statistically  identify  discussions  that  are 
focused  on  concepts  in  novel  combinations?  In  other  words,  are  they  “on 
topic”  but  not  explicitly  addressed  by  the  reference  documents?  These  are 
the  discussions  where  significant  innovation  may  be  occurring. 

•  Improving  the  defense  acquisition  process  is  a  major  challenge  that  holds 
potentially  massive  payoffs.  Decision-milestone  preparations  can  benefit  from 
broader  review  and  judicious  cross-program  comparisons  that  discover 
possibilities  that  aren’t  already  recognized.  Future  rounds  of  the  Bll 
MMOWGLI  game  will  continue  investigating  how  crowd-sourcing  techniques 
might  best  be  applied  to  make  a  good  acquisition  process  even  better. 
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Appendix  A:  Gaps  and  Opportunity  Areas  to  Integrate  the  Innovative  Concepts  and 
Action  Plans  From  the  MMOWGLI  Energy  Game  Into  Current  Navy  Program  Elements 

“Fuel,”  as  an  independent  variable,  can  be  crucial  for  improving  DoD  energy 
efficiency.  For  example,  according  to  the  DoD  energy  inefficiency  report  (DoD  Acquisition 
Technology,  2012), 


The  current  process  either  does  not  consider  fuel,  or  considers  only  the 
commodity  price.  However,  moving  fuel  into  and  around  the  theater  of 
combat  imposes  significant  operational  burdens  and  vulnerabilities,  drives 
force  structure  toward  support  at  the  expense  of  combat  operations,  and 
increases  costs  for  delivery  and  logistics.  Neither  current  requirements  nor 
acquisition  processes  accurately  explore  tradeoff  opportunities  using  fuel  as 
an  independent  variable.  This  prevents  an  end-to-end  view  of  fuel  utilization 
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and  distorts  platform  design  choices,  consequently  preventing  DoD  from 
achieving  maximum  combat  benefit  for  its  logistics  effort. 

We  argue  that  by  matching  the  data  and  consensus  gathered  from  the  collective 
intelligence  platform  (e.g.,  MMOWGLI  energy  game  data  with  the  current  existing  DOD 
programs,  exemplified  in  the  Navy  2013  PEs  documents),  we  can  identify  critical  variables, 
elements,  concepts  or  word  pairs  that  are  linked  to  energy.  Therefore,  without  imposing 
significant  operational  burdens  and  vulnerabilities,  innovative  “energy  efficiency”  ideas  from 
the  game  might  be  naturally  implemented  into  the  current  processes  that  drives  force 
structures,  combat  operations,  delivery,  and  logistics. 

We  use  match  matrices  for  each  individual  theme  found  through  LLA  to  identify 
energy-related  parameters  or  elements  as  word  pairs,  and  then  we  use  these  word  pairs  to 
identify  the  opportunities  in  the  current  process  (i.e. ,  what  PEs  might  be  good  candidates  to 
engage  the  energy-related  parameters/elements/concepts/word  pairs  discussed  in  the 
MMOWGLI  energy  game).  These  findings  are  listed  below. 
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The  match  matrix  for  Theme  430  suggests  that  PEs  mentioned  the  concepts 
“existing  fleet,”  “shipboard  system(s),”  “shipboard  equipment,”  and  “secondary  power”  that 
might  have  the  overall  potential  to  engage  Action  Plans  10,  26,  and  18. 

•  Action  Plan  10:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 

•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 

•  Action  Plan  18:  Offshore  basing. 
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The  match  matrix  for  Theme  393  suggests  that  the  PEs  with  the  concepts  “Navy 
energy,”  “energy  systems,”  “power  generation,”  “alternative  fuel,”  “alternative  energy,” 
“renewable  sources,”  and  “costs — energy/infrastructure”  could  be  used  good  candidates  to 
implement  the  innovative  ideas  related  to  Action  Plans  11,  18,  22,  and  35. 


•  Action  Plan  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet. 

•  Action  Plan  18:  Offshore  basing. 


•  Action  Plan  22:  Scaling  the  small  solutions:  Energy  recycling  and  rethinking 
“The  Big  Fix.” 


The  match  matrix  for  Theme  458  shows  that  the  PEs  mentioned  (“naval 
expeditionary,”  “ship  board,”  and  “strike  carrier”)  can  good  candidates  to  engage  Action 
Plans  15  and  26. 


•  Action  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in  real  time. 
That  way  the  nearest  force  will  response  and  reduce  travel  distances. 

•  Action  26:  Expand  use  of  nuclear  power  in  the  fleet. 

Related  concepts  include  “multiple  hardware,”  “operating  time,”  and  “dashboard 
energy.” 
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The  matrix  for  Theme  905  that  the  PEs  involved  (“unmanned  systems,”  “surface 
ships,”  “nuclear  powered,”  “operational  environment,”  and  “water  treatment”)  can  be  good 
candidates  for  engaging  Action  Plans  18,  19,  20,  26,  31 , 35,  4,  and  7. 


•  Action  Plan  18:  Offshore  basing. 

•  Action  Plan  19:  Implement  self-sustaining  support  infrastructure  on  all  Navy 
bases. 

•  Action  Plan  20:  Sails  on  vessels;  use  sails  that  are  foldable  on  the  sides  of 
vessels. 


•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 

•  Action  Plan  31 :  Add  “reducing  energy  consumption”  to  Battle  E  criteria. 

•  Action  Plan  35:  Create  3D/vertical  farms  for  use  in  growing  biofuels  and  crop 
for  human  consumption. 

•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 

•  Action  Plan  7:  Install  “sea  brakes”  that  generate  electricity,  like  a  Prius.  These 
could  be  used  to  aid  in  docking/slowing  ships  and  reduce  the  need  for  tugs. 
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The  match  matrix  for  Theme  132  shows  that  the  PEs  mentioned  (“additional  energy,” 
“ground  forces”  [e.g.,  PE  06021 31 M,  PE  0603640M;  PE  020631 3M;  PE  0602750N;  PE 
060501 3M;  PE  0604404N],  “harvesting  energy”  [e.g.,  PE  0602236N:  Warfighter 
Sustainment  Applied  Res;  PE  0603673N:  (U)Future  Naval  Capabilities  Advanced  Tech  Dev; 
PE  0601 153N:  Defense  Research  Sciences;  PE  06021 23N:  Force  Protection  Applied  Res], 
“potential  energy,”  and  “hydrodynamic  forces”)  are  the  good  candidates  to  engage  Action 
Plans  14,  15,  17,  18,  34,  and  7. 

•  Action  Plan  14:  Recycle  everything  biological  into  fuel:  waste,  etc. 

•  Action  Plan  15:  A  global  navy  formed  by  an  alliance  of  nation  linked  in  real 
time.  That  way,  the  nearest  force  will  response  and  reduce  travel  distances. 

•  Action  Plan  17:  Energy  harvesting  satellites  in  outer  space  transmit  it  to  Earth 
via  microwave  or  laser  beam. 

•  Action  Plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore  bases 
in  critical  regions  such  as  the  South  China  Sea. 

•  Action  Plan  34:  Create  online  system  or  suggestion  card  system  for  Navy 
personnel  to  input  where  they  see  energy  savings  in  their  job. 

•  Action  Plan  7:  Install  “sea  brakes”  that  generate  electricity,  like  a  Prius.  These 
could  be  used  to  aid  in  docking/slowing  ships,  reduce  need  for  tugs. 
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The  match  matrix  for  Theme  787  suggests  that  the  PEs  (“energy  efficiency”  and  “fuel 
efficiency”)  can  be  viewed  as  “survivability  requirements”;  therefore,  any  PEs  related  to 
“survivability  requirements”  (e.g.,  PE  060321 6N:  Aviation  Survivability)  or  “operational 
requirements”  can  be  used  to  engage  Action  Plans  10,  1 1 , 20,  27,  31 ,  34,  and  9. 

•  Action  Plan  9:  Composite  ship  design:  Explore  the  use  of  polymer  substrates 
for  improved  ship  structural  design. 


•  Action  Plan  10:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 


1 — 

_ 

; 

The  match  matrix  for  Theme  494  suggests  that  the  PEs  mentioned  (“shared 
information,”  “signal  intelligence,”  “share  data,”  “data  structures,”  “intelligence  systems,” 
“artificial  intelligence,”  and  “maritime  warfare”)  might  be  good  candidates  to  engage  Action 
Plans  16,  18,  26,  31,  and  36. 

•  Action  Plan  16:  Using  synthetic  lubricants  to  save  5%  to  25%  of  energy  costs. 

•  Action  Plan  18:  Create  flotillas  of  ships  and  sea  platforms  as  off  shore  bases 
in  critical  regions  such  as  the  South  China  Sea. 

•  Action  Plan  36:  Become  more  efficient  at  structured,  logical  dialogue  to  find 
the  solutions  being  sought. 
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The  match  matrix  for  Theme  633  suggests  that  the  PEs  mentioned  (“advanced  tech” 
[e.g.,  PE  0603673N:  (U)Future  Naval  Capabilities  Advanced  Tech  Dev],  “greater  efficiency” 
[e.g.,  PE  0603747N:  Undersea  Warfare  Advanced  Tech],  and  “power  plants”)  can  be  good 
candidates  to  engage  Action  Plans  11,21,  and  4. 

•  Action  Plan  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet. 

•  Action  Plan  21 :  DoD  shore  facility  energy  independence:  Explore  use  of 
thorium-based  reactors  (liquid  fluoride  thorium  reactor  [LFTR])  for  power 
generation  off  the  grid. 

•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 


The  match  matrix  for  Theme  326  suggests  that  the  PEs  mentioned  (“energy 
security,”  “missile  defense,”  “operational  security,”  “cyber  security,”  “national  security,”  and 
“Naval  Postgraduate  School”)  might  be  good  candidates  to  engage  Action  Plans  17,  19,  4, 
27,  4,  35,  and  5. 

•  Action  Plan  17:  Energy  harvesting  satellites/space-based  solar  power. 

•  Action  Plan  19:  Implement  self-sustaining  support  infrastructure  on  all  Navy 
bases. 

•  Action  Plan  4:  Change  small  land  vehicle  transportation  to  hybrid  vehicles  in 
non-combat  capacity. 
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The  match  matrix  for  Theme  917  suggests  that  the  PEs  mentioned  (“nuclear  power,” 
“nuclear  technology,”  “safety  standards,”  “logistics  systems,”  “logistics  management,” 
“standards  development/data,”  and  “common  standards”)  might  be  good  candidates  to 
engage  Action  Plans  16,  18,  25,  26,  31,  34,  and  9. 

•  Action  Plan  34:  Create  online  system  or  suggestion  card  system  for  Navy 
personnel  to  input  where  they  see  energy  savings  in  their  job. 


The  match  matrix  for  Theme  579  suggests  that  the  PEs  mentioned  (“energy 
management,”  “composite  materials,”  “processing  capabilities,”  “supply  chains,”  “electrical 
energy,”  “hazardous  waste,”  “energy  absorbing,”  “sinks  heat,”  “heat  reduce,”  and  “naval 
academy”)  might  be  good  candidates  to  engage  Action  Plans  8,  20,  26,  and  9. 


•  Action  Plan  8:  Shore  energy  optimization  strategy:  Recommendations  for 
improvements  and  implementation. 


The  match  matrix  for  Theme  854  suggests  that  PEs  mentioned  (“turbine  engine,” 
“diesel  engine,”  “energy  sources,”  “power  sources,”  and  “greenhouse  gas”)  might  be  good 
candidates  to  engage  “behavior  modification”  related  Action  Plans  27,  8,  and  5. 


•  Action  27:  Upgrade  Navy  housing  with  SMART  grids  to  reduce  energy 
consumption.  By  individualizing  electricity/utility  bills  to  single  households, 
family  users  will  be  motivated  to  increase  energy  saving  efforts. 

•  Action  5:  Incentivize  behavior  to  reduce  electricity  usage  in  Navy  housing. 

•  Action  8:  Update  older  buildings  to  be  more  energy  efficient.  The  Navy  is  still 
using  buildings  that  are  almost  a  century  old. 

These  PEs  include,  for  example,  PE  0603573N:  Advanced  Surface  Machinery  Sys; 
PE  0603724N:  Navy  Energy  Program;  PE  0205633N:  Aviation  Improvements;  PE 
0206623M:  MC  Ground  Cmbt  Spt  Arms  Sys;  and  PE  0605864N:  Test  &  Evaluation  Support. 
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The  match  matrix  for  Theme  732  suggests  that  the  PEs  mentioned  (“ship  surface,” 
“fleet  surface,”  “power  management,”  “ship  power,”  “supplying  power,”  and  “generating 
power”)  might  be  good  candidates  to  engage  action  plans  mentioned  (“mobile  power,” 
“electric  warship,”  “training  centers”  and  “ocean  wave”).  These  PEs  include,  for  example,  the 
following: 

•  PE  0603563N:  Ship  Concept  Advanced  Design 

•  PE  06021 23N:  Force  Protection  Applied  Res 

•  PE  0603573N:  Advanced  Surface  Machinery  Sys 

•  PE  0206624M:  Marine  Corps  Cmbt  Services  Supt 

•  PE  06031 14N:  Power  Projection  Advanced  Technology 

•  PE  0601 153N:  Defense  Research  Sciences 

•  PE  0602131 M:  Marine  Corps  Lndg  Force  Tech 
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The  match  matrix  for  Theme  449  suggests  that  the  PE  mentioned  (“power 
projection”)  can  be  used  to  engage  “social  media”  for  “fuel/energy  saving.” 
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•  Action  1 1 :  Enhanced  education  to  develop  an  energy  efficient  fleet,  engage 
major  universities  to  create  a  cross-disciplinary  curriculum  for  “energy  design” 
in  all  fields  for  all  forms  of  energy. 


The  match  matrix  for  Theme  682  suggests  that  the  PEs  mentioned  (“ship 
construction,”  “ship  operations,”  “fleet  operations,”  “military  construction,”  and  “operations 
research”)  can  be  good  candidates  to  engage  Action  Plans  10,  26,  and  6. 


•  Action  Plan  10:  In  this  era  of  convergence,  reduce  the  number  of  shipboard 
systems  and  focus  more  on  small  computers  with  high  capability  (Android, 
iOS  apps). 

•  Action  Plan  26:  Expand  the  use  of  nuclear  power  in  the  fleet  and  ashore. 


•  Action  Plan  6:  Implement  large  umbrellas  for  ships  to  use  shading  to  keep 
ship  cooler;  also  use  “carport”  structures  for  ships  docked  on  the  pier. 
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The  match  matrix  for  Theme  257  suggests  that  the  PEs  mentioned  (“parts 
replacement,”  “communication  equipment,”  “air  wing,”  “communication  data,”  and  “urban 
environments”)  might  be  good  candidates  for  Action  Plans  16,  18,  27,  28,  34,  and  35. 


•  Action  16:  Using  synthetic  lubricants  to  save  5%  to  25%  of  energy  costs. 

•  Action  18:  Offshore  basing. 

•  Action  27:  Upgrade  Navy  housing  with  SMART  grids  to  reduce  energy 
consumption.  By  individualizing  electricity/utility  bills  to  single  households, 
family  users  will  be  motivated  to  increase  energy  saving  efforts. 

•  Action  28:  Power  on-board  minor  electronics  with  stationary  bikes  used  for 
personnel  fitness  training. 

•  Action  34:  Online  feedback  and  social  networking. 

•  Action  35:  3D  farming:  Less  land  use  and  local  agriculture  reducing  fuel  use 
and  potential  location  of  bio-fuel  crops. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  520- 


The  match  matrix  for  Theme  198  suggests  that  the  PEs  mentioned  (“energy  saving,” 
“fuel  savings,”  “cost  savings,”  “fuel  cell,”  “cell  technologies,”  “storage  energy,”  and  “storage 
systems”)  might  be  good  candidates  to  engage  Action  Plans  related  to  these  concepts. 


The  resulted  matrices  from  this  task  will  help  design  the  specific  questions  to 
address  the  issues  on  a  program-to-program  basis  to  continue  the  energyMMOWGLI  game 
with  acquisition  professionals  in  the  acquisition  research  community  in  the  future. 
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Addressing  Counterfeit  Parts  in  the  DoD  Supply  Chain1 

Jacques  S.  Gansler — The  Honorable  Jacques  S.  Gansler,  former  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  is  a  professor  and  holds  the  Roger  C.  Lipitz  Chair  in  Public 
Policy  and  Private  Enterprise  in  the  School  of  Public  Policy,  University  of  Maryland;  he  is  also  the 
director  of  the  Center  for  Public  Policy  and  Private  Enterprise.  As  the  third-ranking  civilian  at  the 
Pentagon  from  1997-2001,  Dr.  Gansler  was  responsible  for  all  research  and  development, 
acquisition  reform,  logistics,  advance  technology,  environmental  security,  defense  industry,  and 
numerous  other  security  programs.  Before  joining  the  Clinton  Administration,  Dr.  Gansler  held  a 
variety  of  positions  in  government  and  the  private  sector,  including  Deputy  Assistant  Secretary  of 
Defense  (Material  Acquisition),  Assistant  Director  of  Defense  Research  and  Engineering 
(Electronics),  senior  vice  president  at  TASC,  vice  president  of  ITT,  and  engineering  and  management 
positions  with  Singer  and  Raytheon  Corporations. 

Throughout  his  career,  Dr.  Gansler  has  written,  published,  testified,  and  taught  on  subjects 
related  to  his  work.  He  is  the  author  of  five  books  and  over  100  articles.  His  most  recent  book  is 
Democracy’s  Arsenal:  Creating  a  21st  Century  Defense  Industry  (MIT  Press,  2011). 

In  2007,  Dr.  Gansler  served  as  the  chair  of  the  Secretary  of  the  Army’s  Commission  on 
Contracting  and  Program  Management  for  Army  Expeditionary  Forces.  He  is  a  member  of  the 
Defense  Science  Board  and  the  Government  Accountability  Office  Advisory  Board.  He  is  also  a 
member  of  the  National  Academy  of  Engineering  and  a  fellow  of  the  National  Academy  of  Public 
Administration.  Additionally,  he  is  the  Glenn  L.  Martin  Institute  Fellow  of  Engineering  at  the  A.  James 
Clarke  School  of  Engineering;  an  affiliate  faculty  member  at  the  Robert  H.  Smith  School  of  Business; 
and  a  senior  fellow  at  the  James  MacGregor  Burns  Academy  of  Leadership  (all  at  the  University  of 
Maryland).  From  2003-2004,  Dr.  Gansler  served  as  interim  dean  of  the  School  of  Public  Policy  at  the 
University  of  Maryland,  and  from  2004-2006,  he  served  as  Vice  President  for  Research  at  the 
University  of  Maryland,  [jgansler@umd.edu] 

William  Lucyshyn — Mr.  Lucyshyn  is  the  Director  of  Research  and  a  senior  research  scholar  at  the 
Center  for  Public  Policy  and  Private  Enterprise  in  the  School  of  Public  Policy  at  the  University  of 
Maryland.  Previously,  Mr.  Lucyshyn  served  as  a  program  manager  and  the  Principal  Technical 
Advisor  to  the  Director  of  the  Defense  Advanced  Research  Projects  Agency  (DARPA)  on  the 
identification,  selection,  research,  development,  and  prototype  production  of  advanced  technology 
projects.  Prior  to  joining  DARPA,  Mr.  Lucyshyn  completed  a  25-year  career  in  the  U.S.  Air  Force.  Mr. 
Lucyshyn  received  his  bachelor’s  degree  in  engineering  science  from  the  City  University  of  New  York 
and  earned  his  master’s  degree  in  nuclear  engineering  from  the  Air  Force  Institute  of  Technology.  He 
has  authored  numerous  reports,  book  chapters,  and  journal  articles.  [Iucyshyn@umd.edu] 

John  Rigilano — Rigilano  is  a  faculty  research  assistant  at  the  Center  for  Public  Policy  and  Private 
Enterprise.  He  earned  his  Master  of  Public  Policy  degree  from  the  University  of  Maryland,  College 
Park,  in  2011  and  holds  a  Bachelor  of  Arts  degree  in  anthropology  from  the  Pennsylvania  State 
University.  He  is  pursuing  a  career  in  policy  and  program  analysis,  [jprig@umd.edu] 

Introduction 

In  June  2007,  the  U.S.  Department  of  the  Navy,  Naval  Air  Systems  Command 
(NAVAIR),  asked  the  Bureau  of  Industry  and  Security’s  (BIS)  Office  of  Technology 
Evaluation  (OTE)  to  conduct  a  defense  industrial  base  assessment  of  counterfeit 
electronics.  NAVAIR  suspected  that  an  increasing  number  of  counterfeit/defective 
electronics  was  infiltrating  the  DoD  supply  chain  and  affecting  weapon  system  reliability. 
OTE  data  revealed  that  39%  of  companies  and  organizations  participating  in  the  survey 
encountered  counterfeit  electronics  during  the  four-year  study  period.  Moreover,  the 


1  This  is  a  summary  of  the  full  report,  which  will  be  available  in  July  2013. 
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frequency  of  detected  counterfeit  incidents  was  escalating  rapidly,  rising  from  3,868 
incidents  in  2005  to  9,356  incidents  in  2008.  These  counterfeit  incidents  included  multiple 
versions  of  DoD  qualified  parts  and  components  (U.S.  Department  of  Commerce,  2010). 

Today,  the  DoD  procures  systems  and  products  from  a  large  network  of  global 
suppliers  and  manages  over  four  million  different  parts  at  a  cost  of  over  $94  billion  (GAO, 
2010).  As  the  DoD  draws  from  this  increasingly  global  supplier  base,  its  visibility  into  these 
source  companies  is  often  limited;  quality  controls  are,  at  times,  insufficient;  and  chain  of 
custody  verification  is  lacking.  As  a  result,  the  challenge  of  assuring  the  integrity  and 
provenance  of  parts  and  components  has  grown  geometrically  more  complex  in  this  global 
sourcing  environment. 

When  they  are  installed  in  systems,  counterfeit  parts  and  components  can  affect  the 
safety,  operational  readiness,  cost,  and  critical  nature  of  the  military  mission.  Almost  any 
part  can  be  counterfeited — including  fasteners  used  on  aircraft,  electronics  used  on  missile 
guidance  systems,  and  materials  used  in  body  armor  and  engine  mounts.  Counterfeit  parts 
have  the  potential  to  cause  a  serious  disruption  to  DoD  supply  chains,  delay  ongoing 
missions,  and  even  affect  the  integrity  of  weapons  systems  (GAO,  2010). 

Additionally,  as  DoD  weapon  systems  age,  products  required  to  support  them  may 
no  longer  be  available  from  the  original  manufacturers  or  through  franchised  or  authorized 
suppliers.  Instead,  the  DoD  must  turn  to  independent  distributors,  brokers,  or  aftermarket 
manufacturers  as  sources  of  supply.  Here  again,  the  DoD  is  at  risk  for  acquiring  counterfeit 
parts. 

At  the  same  time,  counterfeiters  continue  to  develop  more  sophisticated  capabilities, 
making  detection  all  the  more  difficult.  For  instance,  third-party  subcontractors  for  major 
defense  companies  have  been  found  to  manufacture  working  components,  only  to  mix  them 
with  cheaper  parts  of  inferior  quality  and/or  non-working  components.  Needless  to  say, 
schemes  of  this  sort  make  determining  the  provenance  of  counterfeit  components 
exceedingly  difficult. 

Over  the  years,  counterfeiters  have  also  fine-tuned  their  ability  to  replicate  parts, 
often  by  relying  on  scrap  materials  that  were  thought  to  have  been  destroyed  (Martin,  2012). 
The  burgeoning  practice  of  harvesting  and,  often,  repurposing  electronic  waste  or  “e-waste” 
(e.g.,  discarded  computers,  office  electronic  equipment,  entertainment  device  electronics, 
mobile  phones,  telephones,  and  refrigerators)  poses  a  growing  challenge  to  the  DoD.  In  the 
slums  of  China,  India,  and  Pakistan,  peasants  “cook”  circuit  boards  over  trash  can  fires  in 
order  to  remove  the  metal  chips,  selling  them  to  local  counterfeiting  operations.  Once  the 
chips  are  cleaned,  refurbished,  and  relabeled,  they  are  purchased  by  unscrupulous  military 
subcontractors  that  go  on  to  supply  “military  grade”  microchips  to  many  of  America’s  largest 
defense  companies.  According  to  a  2012  GAO  report,  some  of  these  microchips  are  then 
used  in  some  of  the  DoD’s  major  weapons  systems. 

In  this  environment,  the  DoD  must  step  up  its  war  against  counterfeit  parts,  much  as 
private  industry  has  done.  For  example,  counterfeit  drugs  are  rare  (at  least  in  the  United 
States)  thanks  to  the  relatively  high  level  of  safety  assuredness  for  U.S.  pharmaceuticals 
(Lechleiter,  2012).  This  includes  the  review  of  production  yields,  capacity,  and/or  product 
amounts  compared  with  raw  material  purchases.  Given  the  relative  ease  with  which 
authentic-looking  drugs  can  be  reproduced  (indeed,  reproducing  packaging  is  more 
expensive  than  making  the  fake  drug),  it  is  remarkable  that  there  are  so  few  reported 
instances  of  counterfeits. 
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Across  the  DoD  supply  chain,  however,  counterfeits  of  all  types — from  electronic 
equipment  to  metal  fasteners — have  been  found.  As  a  direct  consequence,  the  lives  of 
military  men  and  women  are  at  stake.  Thus  far,  the  impact  of  counterfeit  parts  has  been 
minimal  in  this  regard.  According  to  Pentagon  Press  Secretary,  George  Little,  “[the  DoD]  is 
unaware  to  date  of  any  loss  of  life  or  catastrophic  mission  failure  that  has  occurred  because 
of  counterfeit  parts”  (Garamone,  2012,  p.  1).  But  given  the  growth  of  the  availability  of 
counterfeit  parts,  it  may  only  be  a  matter  of  time. 

All  branches  of  the  Services  are  affected  by  the  threat  of  counterfeit  parts.  The 
following  examples  illustrate  cases  in  which  counterfeit  parts  have  infiltrated  the  Services’ 
supply  chains  (GAO,  2010). 

•  Army:  Seatbelt  clasps.  Seatbelt  parts  were  made  from  a  grade  of  aluminum 
that  was  inferior  to  that  specified  in  the  DoD’s  requirements.  The  parts  were 
found  to  be  deficient  when  the  seatbelts  were  accidentally  dropped  and  they 
broke. 

•  Navy:  Routers.  The  Navy,  as  well  as  other  DoD  and  government  agencies, 
purchased  counterfeit  network  components — including  routers — that  had  high 
failure  rates  and  the  potential  to  shut  down  entire  networks.  A  two-year  FBI 
criminal  investigation  led  to  10  convictions  and  $1.7  million  in  restitution. 

•  Air  Force:  Microprocessor.  The  Air  Force  needed  microprocessors  that 
were  no  longer  produced  by  the  original  manufacturer  for  its  F-1 5  flight- 
control  computer.  These  microprocessors  were  procured  from  a  broker,  and 
F-1 5  technicians  noticed  additional  markings  on  the  microprocessor  and 
character  spacing  inconsistent  with  the  original  part.  A  total  of  four  counterfeit 
microprocessors  were  found  and  as  a  result  were  not  installed  on  the  F-15’s 
operational  flight  control  computers. 

•  Defense  Logistics  Agency:  Packaging  and  small  parts.  During  a  two-year 
period,  a  supplier  and  three  co-conspirators  packaged  hundreds  of 
commercial  items  from  hardware  and  consumer  electronics  stores  and 
labeled  them  as  military-grade  items.  For  example,  a  supplier  labeled  the 
package  containing  a  circuit  from  a  personal  computer  as  a  $7,000  circuit  for 
a  missile  guidance  system.  The  suppliers  avoided  detection  by  labeling 
packages  to  appear  authentic,  even  though  they  contained  the  wrong  part. 
The  supplier  received  $3  million  from  contracts  totaling  $8  million  before 
fleeing  the  country. 

Defense  contractors  are  encouraged  to  report  counterfeits  using  the  Government 
Industry  Data  Exchange  Program  (GIDEP)  database.  GIDEP  serves  as  a  data  repository  for 
the  collection  and  sharing  information  on  nonconforming  parts  and  materials,  including 
information  on  suspect  counterfeit  products,  allowing  government  and  industry  participants 
to  share  information.  However,  not  all  participants  are  willing  to  share  such  information.  This 
is  not  surprising  given  the  lack  of  clear  incentives,  especially  if  the  participating  firm  believes 
their  reputation  may  be  damaged  as  a  result. 

In  order  to  reduce  the  risk  of  counterfeit  parts  in  the  DoD  supply  chain,  we  provide 
the  following  high-level  recommendations: 

•  Partner  with  industry  to  develop  a  network  of  trusted  providers. 

•  Mandate  that  suppliers  report  suspect  counterfeits  using  GIDEP  and  provide 
penalties  for  non-compliance. 
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•  Require  that  the  supplier  absorb  any  costs  associated  with  the  removal  and 
replacement  of  counterfeit  parts  or  components  that  make  their  way  into  DoD 
systems. 

•  Invest  in  visibility  systems  to  track  the  provenance  and  transport  of  parts  and 
components. 

•  Adopt  regionalized  supply  chains  to  reduce  supplier  and  transport  risk. 

The  threat  of  counterfeit  parts  within  the  DoD’s  supply  chain  is  real  and  will  only 
escalate  over  time,  with  potentially  serious  consequences.  In  order  to  reduce  this  threat,  the 
DoD  and  its  industry  partners  will  have  to  work  together.  While  both  may  have  the  best 
intentions,  it  is  essential  that  incentives,  penalties,  and  rewards  are  properly  aligned  in  order 
to  produce  the  desired  outcome. 
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Abstract 

Using  the  Defense  Logistics  Agency’s  current  service  performance  metric  Next  Scheduled 
Departure,  we  develop  methodologies  for  establishing  the  optimal  timing  of  order  releases  in 
a  distribution  center  so  that  customers  receive  supplies  sooner.  We  present  a  simulation 
model  to  test  these  methodologies  and  to  show  that  setting  wave  release  times  accordingly 
can  significantly  improve  service  performance  for  systems  subject  to  stationary  and  non¬ 
stationary  arrivals. 

Introduction 

Continuing  fiscal  struggles  in  the  federal  government  have  made  “do  more  with  less” 
the  operating  mode  of  almost  every  Department  of  Defense  (DoD)  organization.  The 
Defense  Logistics  Agency  (DLA)  and  its  distribution  centers  are  no  exception.  In  DLA’s 
case,  there  is  increasing  pressure  to  provide  superior  service  with  the  same  or  fewer  labor 
resources.  By  “superior  service,”  we  mean  rapid  response  to  customer  requisitions. 

“Operational  availability”  (/40)  of  a  system  is  defined  as  the  fraction  of  time  or 
probability  that  a  system’s  capabilities  will  be  available  for  operational  use  (“Operational 
Availability  Handbook,”  2003).  A0  is  a  function  of  “uptime”  and  “downtime,”  the  latter  being 
mainly  determined  by  Mean  Logistics  Delay  Time  (MLDT).  Intuitively,  reducing  the  flow 
time — the  time  between  arrival  of  the  order  and  the  time  it  is  ready  to  ship — decreases 
MLDT  and  therefore  improves  A0.  In  general,  the  more  quickly  the  logistics  system  responds 
to  the  requests,  the  higher  the  availability  of  end  items  because  total  downtime  is  reduced 
through  reduced  MLDT. 

If  a  part  is  in  stock,  logistics  delay  time  is  comprised  of  two  main  components: 
warehouse  processing  time  and  transportation  time.  These  two  processes  (warehousing 
and  transportation)  meet  at  the  shipping  dock.  Doerr  and  Gue  (2011)  observed  that 
warehouse  operations  are  effectively  “continuous,”  in  that  completed  orders  arrive  at  a 
shipping  dock,  more  or  less,  in  a  continuous  stream.  By  contrast,  transportation  is  a  cyclical 
process,  due  to  the  need  to  achieve  economies  of  scale.  Thus,  for  example,  package 
carriers  such  as  UPS  and  FedEx  have  a  “nightly  sort”  and  “next  day”  deliveries.  Less-than- 
truckload  (LTL)  carriers,  which  transport  larger  shipments,  also  operate  according  to  a  daily, 
cyclical  model. 


1  This  research  has  been  funded  under  NPS-BAA-1 1  -002  at  Acquisition  Research  Program  at  the 
Naval  Postgraduate  School.  Award  Period:  November  27,  2011-November  26,  2012. 
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To  coordinate  the  internal,  continuous  operations  of  its  DCs  with  the  cyclical 
transportation  schedules  of  its  transportation  providers,  DLA  uses  a  metric  called  Next 
Scheduled  Departure  (NSD),  which  measures  the  fraction  of  orders  arriving  during  a 
specified  24-hour  period  that  are  processed  before  a  specific  truck  departure  (Doerr  &  Gue, 
2011).  By  definition,  an  increase  in  the  metric  means  more  orders  make  their  last  departing 
trucks,  and  that  some  customers  receive  their  orders  before  they  otherwise  would  have.  For 
repair  parts  and  mission-critical  consumables,  therefore,  increasing  NSD  reduces  MLDT  and 
increases  operational  readiness. 

One  would  think  that  making  an  order  available  to  pickers  as  soon  as  it  arrives  would 
increase  its  chance  of  making  it  onto  the  next  departing  truck,  but  such  a  view  misses  the 
economies  of  scale  in  a  picking  operation.  A  worker  picking  a  large  batch  is  much  more 
efficient  than  a  worker  picking  a  single  order,  and  therefore  his  capacity  is  higher.  Higher 
capacity  reduces  waiting  for  arriving  orders,  and  therefore  tends  toward  lower  total  sojourn 
times.  The  benefits  of  large  batches,  however,  must  be  weighed  against  the  queueing  time 
necessary  to  form  the  batch.  To  strike  this  balance,  DCs  at  DLA  release  orders  in  large 
batches  called  waves. 

Despite  the  ubiquity  of  wave  operations  in  commercial  (and  military)  warehouses, 
there  are  no  analytical  models  to  determine  the  optimal  number  and  timing  of  these  waves, 
especially  to  maximize  performance  against  deadline-oriented  metrics  such  as  NSD,  which 
is  used  at  DLA.  (A  thorough  review  of  literature  is  given  in  Qeven  and  Gue,  2013.)  The  goal 
of  our  work  is  to  improve  the  service  performance  and  thus  MLDT  of  a  distribution  center 
(operated  by  DLA,  the  Services,  or  a  third  party)  by  properly  setting  wave  release  times. 

In  the  next  section,  we  discuss  fulfillment  operations  at  DLA  Distribution  Center, 
Susquehanna,  PA,  a  fulfillment  center  operated  by  DLA  Distribution.  We  analyze  its  order 
flow  data  and  the  current  wave  release  strategy.  In  the  section  Optimal  Wave  Release 
Policies,  we  introduce  our  approximation  models  and  use  those  models  to  verify  a 
simulation  model.  We  discuss  the  details  of  the  simulation  model  in  the  Simulation  Study 
section  and  summarize  our  findings  in  our  conclusion. 

Wave  Operations  at  DLA  Distribution 

DLA  Distribution  Susquehanna,  Pennsylvania  (hereafter,  DDSP)  is  an  extremely 
complex  distribution  operation  handling  more  than  one  million  stock-keeping  units  (SKUs) 
stored  among  dozens  of  warehouses.  One  of  its  main  service  offerings  is  Dedicated  Truck, 
in  which  a  customer  requests  specific  times  of  delivery  each  day  or  week.  Delivery  times 
could  be,  for  example,  daily  at  1600,  or  every  Tuesday  and  Friday  at  1200,  depending  on 
the  customer.  DDSP  then  establishes  departure  times  for  trucks  leaving  to  each  Dedicated 
Truck  customer. 

Because  an  order  arriving  just  before  truck  departure  cannot  possibly  be  processed 
in  time,  DDSP  establishes  an  internal  cutoff  time  (or  set  of  cutoff  times,  as  appropriate)  for 
each  Dedicated  Truck  customer.  Orders  arriving  before  that  time  are  due  on  the  appropriate 
“next  truck”  and  must  be  processed  before  it  departs.  Distribution  centers  in  the  DLA 
measure  their  service  performance  with  NSD,  which  measures  the  percentage  of  orders 
arriving  between  consecutive  cutoff  times  that  make  it  on  the  assigned  truck.  Figure  1 
illustrates  how  the  metric  works. 
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Figure  1.  Orders  Arriving  Between  Consecutive  Cutoff  Times  Are  Due  on 

the  Next  Deadline 


Managing  the  release  of  work  to  the  system  in  such  an  environment  is  a  difficult  task, 
to  say  the  least,  and  especially  so  in  the  presence  of  waves.  In  a  typical  distribution 
operation,  including  at  DDSP,  there  are  2-6  waves  per  day,  depending  on  the  workload  and 
number  of  destinations  that  must  be  served.  Figure  2b  shows  the  scheduled  wave  release 
times  at  DDSP  at  the  time  of  our  study — 0000,  0400,  0930,  and  1600.  In  addition  to  these 
scheduled  releases,  orders  were  occasionally  released  manually  at  around  0700  and  0900 
to  balance  the  workload. 


Figure  2.  Number  of  Orders  Arrived  and  Released  Within  a  Day 


It  is  our  experience  that  the  number  and  timing  of  order  releases  is  based  on  intuition 
and  experience  of  management.  Could  NSD  be  improved  if  the  release  times  were 
changed?  What  level  of  benefit  is  possible?  Before  looking  at  the  details  of  DDSP,  we 
present  an  overview  of  mathematical  models  to  establish  order  release  times. 

Optimal  Wave  Release  Policies 

We  first  discuss  some  major  results  from  Qeven  and  Gue  (2013),  in  which  the 
arrival  process  is  assumed  to  be  stationary  with  rate  A  orders  per  unit  time.  The  authors 
propose  a  fluid  approximation  model  in  which  individual  orders  are  indistinguishable.  (They 
also  specify  in  which  conditions  this  approximation  is  valid.)  To  maintain  stability,  the 
server’s  capacity  is  assumed  to  be  ^  >  A. 

Arriving  orders  accumulate  in  a  Warehouse  Management  System  (WMS)  virtual 
queue  until  the  next  wave  is  released,  at  which  time  the  quantity  of  orders  in  that  wave 
decreases  at  rate  p  until  the  wave  is  complete.  Waves  in  this  model  are  not  allowed  to 
overlap;  that  is,  the  current  wave  must  be  complete  before  a  new  wave  can  begin.  While  the 
server  is  working  a  wave,  orders  in  the  next  wave  accumulate,  and  the  cycle  continues.  The 
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cutoff  time  for  accepting  orders  is  assumed  to  be  equal  to  the  truck  departure  time,  which  is 
a  worst  case  in  terms  of  NSD.  This  assumption  is  not  necessary,  but  it  makes  the 
presentation  much  easier.  Qeven  and  Gue  (2013)  discuss  how  to  assign  a  realistic  cutoff 
time. 


NSD  in  a  single-wave  system  is  a  function  of  the  wave  release  time  w1,  the  arrival 
rate  A,  and  the  server  capacity  p.  By  definition, 


NSD  = 


#  orders  worked  today  that  arrived  today 
#  orders  that  arrived  today 


(1) 


When  there  is  a  single  class  of  orders  (which  is  a  simplest  version  of  DDSP’s  original 
problem),  the  server  should  finish  the  wave  exactly  at  the  deadline,  and  therefore  w1  —  l  — 
p.  Thus,  the  optimal  NSD  for  a  single  wave  system  is  —  NSD*  —  1  —  p.  This  proposition 
alone  provides  important  insights.  First,  the  server  should  begin  the  work  as  late  as 
possible  in  order  to  allow  as  many  orders  as  possible  to  make  it  into  the  wave.  Second,  the 
wave  should  finish  exactly  at  the  deadline.  Furthermore,  the  result  suggests  that  the  optimal 
cutoff  time  equals  the  optimal  wave  release  time  w1  —  1  —  p,  for  which  NSD  would  be 
100%.  This  can  also  be  argued  intuitively:  releasing  the  wave  before  the  cutoff  time  means 
some  orders  arrive  after  the  release  but  before  the  cutoff.  These  orders  will  certainly  miss 
the  truck.  Releasing  the  wave  after  the  cutoff  time  means  some  orders  are  in  the  wave  but 
are  not  due  on  the  next  truck,  which  reduces  system  capacity  for  the  orders  that  need  to  be 
processed  immediately.  Neither  condition  should  be  optimal,  as  the  result  shows 
mathematically.  Qeven  and  Gue  (2013)  also  address  the  multiple  wave  systems  and 
determine  the  closed  form  optimal  wave  release  times  for  a  system  with  stationary  arrivals: 


Wj  = 


7-i 

N 


1  - 


pj -pN+l 
1  -pN 


,  for  A  —  p 
,for  A  <  p. 


(2) 


The  authors  observe  that  the  first  release  time  does  not  change,  but  later  wave  times 
adjust  as  the  number  of  waves  increases.  As  expected  wN  —  1  —  p  when  N  —  1,  and  wN 
(and  NSD)  converges  to  1  as  the  number  of  waves  N  ->  oo.  That  is,  NSD  improves  as  more 
waves  are  released,  especially  when  expected  utilization  is  high.  As  utilization  increases, 
the  equation  suggests  that  the  maximum  possible  NSD  decreases,  converging  to  ( N  — 
l)/iV.  The  models  presented  in  Qeven  and  Gue  (2013)  can  also  be  extended  to  reflect 
uncertainty  in  both  daily  workload  and  capacity  uncertainty  as  well  as  daily  non-stationary 
arrivals.  Although  their  results  provide  insight  into  the  importance  of  setting  proper  wave 
release  times,  they  only  partially  address  the  problem  faced  by  DDSP.  This  is  because 
Dedicated  Truck  operations  are  only  a  portion  of  each  day’s  workload  at  DDSP,  so  it  is 
impossible  to  assess  capacity  devoted  to  these  orders.  Another  reason  is  the  fact  that 
DDSP  often  receives  orders  days  in  advance  of  when  they  are  scheduled  to  ship,  and  many 
orders  remain  in  queue  until  near  their  deadline.  Another  complication  that  is  not  addressed 
by  Qeven  and  Gue  (2013)  is  the  existence  of  multiple  deadlines.  Nevertheless,  the  results  in 
Qeven  and  Gue  (2013)  provide  us  with  the  ability  to  simulate  and  test  different  wave  release 
policies. 


Simulation  Study 

Using  a  data  set  from  DDSP,  we  analyzed  the  existing  order  arrival  stream  and  wave 
release  policy  in  order  to  generate  input  for  the  analytical  models.  Distribution  centers  of 
DLA  typically  have  outbound  processes  that  include  picking,  packing,  order  consolidation, 
and  shipping.  Example  flow  timing  data  is  given  in  Table  1 . 
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Table  1.  Order  Flow  Timing  Data  Sample 


OD  ORD 

SD  DT 

SD  Tl 

MIT  DT 

MIT  Tl 

LS  DT 

LS  Tl 

ICK  DT 

ICK  Tl 

ACK  DT 

ACK  Tl 

FFER  DT 

FFER  Tl 

M 

ISSION  DT 

ISSION  Tl 

8002R005 

010061 

35900 

010060 

21206 

010060 

22633 

010060 

35404 

010060 

53610 

010060 

61740 

2 

010060 

01700 

8002R007 

010061 

35900 

010060 

21207 

010060 

22633 

010060 

35052 

010060 

53737 

010060 

61740 

2 

010060 

01700 

The  first  entry  in  Table  1  refers  to  the  order  ID.  All  date  and  time  fields  are  defined  as 
Julian  day  and  military  time,  respectively  (e.g.,  2010257  refers  to  September  17,  2010; 
230140  refers  to  time  2301  and  40  seconds).  The  following  two  fields  refer  to  the  scheduled 
departure  time  followed  by  the  arrival  date  and  time  of  the  order.  The  field  RLS  refers  to  the 
date  and  time  that  the  order  is  released  for  picking.  Pick  completion  date  and  time  is  given  in 
PICK  DT  and  PICK  Tl.  The  completion  date  and  time  of  packing  is  given  in  fields  PACK  DT 
and  PACK  Tl.  Because  orders  wait  for  consolidation,  there  is  a  consolidation  date  and  time 
stamp  (given  with  OFFER  DT  and  OFFER  Tl).  The  last  two  data  fields  correspond  to  the 
actual  shipment  date  and  time.  Figure  3  is  an  illustration  of  order  flow. 


Arrival  Release  Pick  done  Pack  done  Offered  Shipped 


waiting  picking  packing  consolidation  waiting 

Figure  3.  Timeline  of  an  Order  Through  Arrival-to-Ship  Process 

We  were  provided  with  three  months  of  order  flow  data  from  January  2010  to  March 
2010  for  Dedicated  Truck  operations  (DTK) — a  total  of  402,406  orders.  Of  those  orders, 
351,866  arrived  in  2010  (87.44%  of  total)  and  351,530  (87.36%  of  total)  orders  were 
shipped  during  the  three-month  interval  and  were  the  subject  of  our  analysis. 

The  managers  of  DDSP  reported  that  the  overall  system  performance  in  NSD  was 
around  72%  over  the  three-month  period  (75.0%  in  January,  57.9%  in  February,  and  70.4% 
in  March  201 1 ).  We  observe  that  NSD  is  highly  variable  throughout  the  length  of  study, 
within  a  range  of  [22.7%,  100%].  On  average,  72%  of  the  customer  orders  were  fulfilled  by 
their  deadline;  however,  on  some  days  NSD  dropped  below  60%  (see  Figure  4). 

Before  describing  the  simulation,  we  must  cover  one  last  detail.  Figure  2  shows  that 
the  arrival  stream  to  DDSP  is  highly  non-stationary.  Qeven  and  Gue  (2013)  show  how  to 
modify  the  basic  wave  release  model  to  handle  this  case.  The  “discretized”  version  of  their 
model  assumes  the  arrival  rate  in  a  discrete  period  of  time  (in  this  case,  15  minutes)  is 
stationary,  but  that  the  mean  rate  may  change  from  hour  to  hour. 

Below  we  discuss  both  intuitive  and  optimal  wave  release  policies  and  show  how 
proper  release  times  can  improve  NSD.  The  simulation  study  serves  two  purposes:  (1)  to 
verify  the  analytical  models  of  Qeven  and  Gue  (2013),  and  (2)  to  demonstrate  that  service 
performance  can  be  improved  with  proper  wave  release  times. 
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Figure  4.  Recorded  Daily  NSD  and  DDSP 

We  model  the  order  fulfillment  system  as  a  three-stage  queuing  system 
corresponding  to  the  picking,  packing,  and  shipping  processes.  We  assume  20  servers  per 
stage  and  identical  exponential  processing  time  distributions  in  each  stage.  This  choice  is 
arbitrary,  of  course,  but  in  the  absence  of  real  data  (DLA  does  not  collect  processing  time 
data),  we  had  no  justification  for  another  choice.  Arriving  orders  are  stored  in  a  virtual  queue 
and  released  in  the  next  wave.  Once  an  order  is  released  for  picking,  available  workers  start 
picking  orders.  Completed  orders  are  sent  directly  to  packing  and  then  to  shipping.  Because 
daily  workload  at  DDSP  varies,  we  test  different  levels  of  utilization  p  =  0.5,  0.75,  0.95.  We 
adjust  the  (exponential)  processing  rate  to  maintain  the  appropriate  utilization.  We  assume 
four  waves  per  day,  as  in  the  operations  at  DDSP  at  the  time  of  the  study. 

Before  applying  different  wave  release  policies,  we  verify  the  simulation  model  by 
comparing  simulated  NSD  with  NSD  according  to  the  analytical  models.  Using  a  stationary 
arrival  stream,  we  determine  the  optimal  release  times  for  a  single  class,  four-wave  system 
for  each  utilization  level.  Optimal  release  times  suggest  NSD  would  be  96.7%,  88.6%,  and 
78.1  %  for  p  =  0.5,  0.75,  0.95.  We  insert  the  release  times  into  the  Simio  simulation  software 
and  run  the  model  for  30  simulated  days,  with  three  days  of  warmup  and  100  replications. 
Fiqure  5  shows  the  results.  The  analytical  model  approximates  the  corresponding  system’s 
NSD  within  1%. 


Figure  5.  Verification  of  the  Simulation  Model 

Note.  The  red  line  indicates  the  approximated  NSD  by  the  analytical  model.  Black,  blue,  and  green 
data  points  correspond  to  the  average  simulated  NSD  for  p  -  0.5,  0.75,  0.95,  respectively. 
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Consistent  with  the  results  in  Qeven  and  Gue  (2013),  average  NSD  drops  as 
utilization  of  the  system  increases.  We  also  observe  that  the  variability  in  NSD  for  both 
policies  increases  as  p  increases. 

Next,  we  consider  a  non-stationary  arrival  stream  representative  of  the  DDSP  data, 
but  we  scale  the  arrival  rates  to  achieve  an  appropriate  utilization.  We  first  test  an  intuitive 
policy  in  which  each  wave  has  the  same  wave  length.  Because  the  system  will  be  busy 
p  —  1  —  of  the  time,  an  equal  time  policy  divides  this  interval  into  four  equal  waves. 

To  test  the  analytical  model,  we  use  the  same  non-stationary  arrival  data  and 
determine  optimal  release  times  and  NSD*.  We  insert  the  optimal  release  times  into  the 
simulation  model  and  estimate  the  NSDe.  Table  2  shows  the  release  times  for  the  optimal 
and  equal  time  policies. 

Table  2  shows  the  approximated  NSD*  of  the  analytical  model.  Similar  to  our  results 
for  stationary  arrivals,  the  simulation  results  are  close  to  the  approximations  (e.g.,  the 
approximation  overestimates  the  NSD  by  around  3%).  The  optimal  policy  performs  9.6%, 
5.9%,  and  1.2%  better  than  the  intuitive  equal  time  policy  for  different  levels  of  utilization. 
Recall  that  the  model  suggested  more  evenly  distributed  releases  as  utilization  increases. 
We  observe  this  situation  especially  for  p  =  0.95. 


Table  2.  Simulation  Results  for  Non-Stationary  Arrivals 


p  =  0.5 

W[ 

w2 

w3 

w4 

NSD* 

(%) 

NSDe(%) 

Optimal  policy 

12:00 

18:05 

21:18 

22:53 

96.4 

93.3 

Equal  time  policy 

12:00 

15:00 

18:00 

21:00 

- 

83.7 

p  =  0.75 

W[ 

w2 

w3 

w4 

NSD* 

(%) 

NSDe(%) 

Optimal  policy 

06:00 

12:36 

17:45 

21:18 

87.3 

85.2 

Equal  time  policy 

06:00 

10:30 

15:00 

19:30 

- 

79.3 

p  =  0.95 

W[ 

w2 

w3 

w4 

NSD* 

(%) 

NSDe(%) 

Optimal  policy 

01:12 

07:30 

12:47 

18:24 

78.3 

75.9 

Equal  time  policy 

01:12 

06:54 

12:36 

18:18 

- 

74.7 

Conclusions 

In  this  study,  we  have  addressed  order  release  problems  in  order  fulfillment  systems 
and  shown  that  setting  wave  release  times  properly  can  improve  NSD,  and  thus  the 
operational  availability  of  supported  end  items.  In  order  fulfillment  systems  such  as  those 
operated  by  DLA,  order  releases  should  be  timed  to  accommodate  daily  deadlines. 

In  a  simulation  study,  we  verified  the  analytical  results  in  Qeven  and  Gue  (2013)  with 
both  stationary  and  non-stationary  arrival  streams.  We  implemented  those  models  to  test 
optimal  policies  against  an  intuitive  policy  and  showed  that  releasing  waves  optimally 
improves  NSD.  Although  the  complexity  of  operations  at  DDSP  made  direct  analysis 
prohibitive,  our  results  suggest  that  NSD  could  be  improved  with  further  investigation  into 
the  number  and  timing  of  order  releases. 
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Abstract 

The  most  basic  representation  of  a  supply  chain  has  three  elements:  supply,  demand,  and 
the  flow  between  the  two.  A  humanitarian  response  supply  chain  (RSC)  has  to  a  large  extent 
unknown  demand  and  at  best  uncertain  supply  demand  with  disruptive  flow.  A  self-sustaining 
supply  chain  (SSSC)  requires  that  the  supply  chain  itself  provide  all  resources  consumed 
while  transporting  supplies,  thus  complicating  the  operations  with  numerous  challenges  and 
unfamiliar  issues.  If  an  RSC  is  self-sustaining,  it  will  reduce  some  of  the  uncertainties  in 
supply.  However,  self-sustaining  response  supply  chains  (SSRSC)  generate  significant 
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additional  cost  for  being  extreme  supply  chains.  To  understand  the  costs  associated  with 
SSRSC  observed  in  special  operations  and  humanitarian  assistance  and  disaster  relief 
(HADR),  they  must  be  compared  and  contrasted  against  the  known  characteristics  of 
traditional  supply  chains.  This  work  explores  the  issues  and  challenges  of  SSRSC  that  arise 
in  logistics  networks. 

Summary 

We  can  partition  supply  chains  into  three  broad  categories  in  terms  of  how  the  supply 
chain  manages  material — traditional,  sustainable,  and  self-sustainable.  Traditional  supply 
chains  that  function  in  traditional  logistics  networks  have  been  well  studied  in  operations 
management.  Sustainable  supply  chains  (SSC)  have  received  considerable  attention  in  this 
age  of  green  consciousness  and  fiscal  austerity,  and  they  can  be  measured  by  looking  at 
their  agility,  adaptability,  and  alignment  (Lee,  2004).  SSC  often  achieve  these  through  “The 
Three  R’s”:  reduction,  reuse,  and  recycling — the  latter  two  “R’s”  are  often  performed  outside 
of  the  traditional  supply  chain.  Self-sustaining  supply  chains  (SSSCs)  extend  themselves 
beyond  the  reuse  and  recycling.  They  require  that  all  resources  consumed  while 
transporting  supplies  to  their  destinations  be  provided  via  the  network  itself.  This  makes 
SSSCs  even  more  complex — they  essentially  become  supply  chain  islands,  where  the 
network  must  be 

•  nimble  enough  to  transport,  create,  conserve,  and  consume  supply; 

•  flexible  enough  to  repair  and  reuse  the  waste  that  it  produces;  and 

•  rigid  enough  to  fulfill  the  ultimate  demand  of  the  supply  chain  while 
simultaneously  fulfilling  its  own  needs  during  the  process  of  delivering  the 
good  or  service  it  promises. 

In  addition  to  the  management  of  material,  one  important  aspect  of  supply  chains  is 
the  environment  in  which  they  operate.  At  one  end  of  the  spectrum  are  traditional  supply 
chains  with  less  variable  fluctuations  in  demand;  on  the  other  end  are  “response”  supply 
chains  in  which  supply,  demand,  customers,  and  network  configurations  continuously 
change  due  to  unpredictability.  A  supply  chain  in  its  most  basic  form  encompasses  three 
elements:  supply,  demand,  and  flow — flow  being  the  intermediary  between  the  other  two 
components.  Typically,  a  traditional  supply  chain  supplies  a  pre-established,  standardized 
product  to  customers  to  meet  a  relatively  constant  and  forecasted  demand  through 
structured  resources  and  continuous  flow.  In  contrast,  at  any  given  time,  a  humanitarian 
response  supply  chain  supplies  a  wide  range  of  products  and  services  fulfilling  spurts  of 
demand  while  sharing  the  flow  and  capacity  with  other  relief  items  (Apte,  2009).  Traditional 
supply  chain  models  may  fail  when  they  are  stressed  due  to  unknowns  and  uncertainties. 
Supply  chains  stressed  in  this  way — extreme  supply  chains — need  special  attention  from  the 
researchers. 

Sustainable,  self-sustainable,  and  response  supply  chains  are  becoming  more 
relevant  to  the  Department  of  Defense  (DoD)  as  we  proceed  through  the  21st  century.  A 
major  reason  for  this  is  the  era  of  fiscal  austerity  that  we  have  entered  after  the  2008 
financial  crisis.  The  U.S.  DoD  budget  is  tighter,  so  it  must  be  able  to  maintain  the  same 
capabilities  as  in  the  past  while  using  fewer  resources.  Thus,  sustainability  and  self¬ 
sustainability  become  key  strategic  initiatives  for  the  DoD.  Strategy  also  comes  into  play 
when  developing  response  supply  chains — as  people  move  to  more  disaster-prone  areas  of 
the  world,  the  U.S.  military  will  continue  to  play  a  major  role  in  being  the  first  responders  in 
humanitarian  assistance  and  disaster  relief  (HADR).  Also,  the  face  of  conflict  has  tacked 
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towards  more  irregular  enemies  vice  large  armed  forces,  thus  requiring  smaller,  independent 
teams  that  coexist  with  each  other  over  long  periods  of  time. 

Virtually  every  supply  chain,  regardless  of  whether  it  is  self-sustaining  or  not,  shares 
common  characteristics  such  as  supply,  inventory,  distribution  networks,  flows,  lead  times, 
information  systems,  customers,  demands,  and  key  performance  indicators.  In  this  research 
we  study  the  similarities  and  differences  of  SSRSC  with  SSSC  to  expose  the  challenges  in 
SSRSC  in  terms  of  operations.  We  believe  studying  operations  is  the  first  step  to 
understanding  the  burden  of  cost  in  such  supply  chains  (Regnier  &  Nussbaum  2011a, 
2011b). 

When  a  self-sustaining  supply  chain  is  initiated,  it  is  endowed  with  a  certain  set  of 
goods.  These  goods  are  used  to  sustain  the  SSSC  itself  during  the  transport,  and  these 
may  also  be  the  same  types  of  items  that  it  is  attempting  to  deliver  to  its  customers.  When  a 
self-sustaining  supply  chain  begins,  it  has  a  limited  amount  of  space  to  carry  all  of  the  goods 
being  delivered  and  consumed  during  the  delivery.  Therefore,  the  choice  of  goods  to  carry 
is  critical  in  that  the  SSSC  cannot  restock  during  the  delivery  process.  The  carriers  must  be 
efficient,  innovative,  and  sustainable  in  their  use  of  goods — they  must  have  the  tools  to  not 
only  reach  their  destination,  but  also  to  have  the  provisions  that  the  customer  desires.  If  the 
supply  chain  runs  out  of  goods,  not  only  does  the  customer  not  receive  goods,  but  also  the 
supply  chain  itself  could  perish,  resulting  in,  at  best,  unfulfilled  demand — at  worst,  loss  of 
life.  Thus,  efficient  reuse  is  critical,  as  space  is  at  a  premium.  Furthermore,  the  logistics 
network  in  an  SSSC  could  be  unstable  and  variable  over  time.  These  SSSCs,  especially 
within  the  context  of  HADR  and  DoD  special  operations,  provide  for  a  unique  research 
opportunity  that  has  not  been  thoroughly  studied  (see  Figure  1). 

For  example,  consider  the  supply  chain  of  providing  fuel.  Transportation  of  this  single 
commodity  requires  fuel  to  be  consumed  by  vehicles  that  transport  it.  There  exist  numerous 
challenges  in  this  network  if  it  is  to  be  self-sustaining.  It  has  also  been  researched  and 
proved  that  such  SSSCs  can  incur  significantly  higher  costs  than  traditional  networks 
(Regnier,  Simon,  &  Nussbaum,  2012).  We  in  this  project  will  study  such  challenges  in 
response  supply  chains,  where  multiple  goods  are  conveyed  and  consumed  through  the 
same  network — a  network  that  is  rife  with  uncertainty. 
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Supply  Chains 


Figure  1.  Positioning  Self-Sustaining  Response  Supply  Chains 

Uncertainty  in  response  supply  chains  is  typified  by  “unknown  unknowns,”  to  quote 
Donald  Rumsfeld.  When  disaster  strikes,  authorities  do  not  know  who  is  hurt,  what  the 
severity  of  the  damage  is,  what  portions  of  the  network  remain  or  are  degraded,  how  the 
supply  chain  will  develop  in  the  future,  where  demand  will  materialize,  where  supply  will 
materialize — to  name  a  few  unknowns.  The  only  knowns  are  the  goals  to  save  lives  and  to 
reduce  suffering.  Saving  lives  involves  delivering  the  goods  that  are  needed  to  sustain  life — 
the  same  goods,  such  as  water,  fuel,  medical  supplies,  equipment,  and  information,  that 
SSSCs  use  and  deliver  during  their  life  cycles.  In  such  instances,  the  transportation 
capabilities  needed  to  deliver  goods,  save  lives,  and  reduce  suffering  have  to  be  reliable. 

The  uncertainties  are  brought  on  due  to  the  process  of  providing  relief  that  in  turn  makes 
SSRSC  more  complex.  We  plan  to  explore  these  complexities  in  SSRSC,  thus  allowing  the 
DoD  to  identify  the  burden  of  cost. 
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Abstract 

This  paper  introduces  a  mixture  distribution  approach  to  modeling  the  probability  density 
function  for  lead  time  demand  (LTD)  in  problems  where  a  continuous  review  inventory  system 
is  implemented.  The  method  differs  from  the  typical  “moment-matching”  approach  by 
focusing  on  building  up  an  accurate,  closed-form  approximation  to  the  LTD  distribution  from 
its  components  by  using  mixtures  of  truncated  exponential  (MTE)  functions.  First, 
construction  of  the  LTD  is  illustrated  and  the  approach  is  compared  to  two  other  possible 
LTDs.  This  distribution  is  then  utilized  to  determine  optimal  order  policies  in  cases  where  a 
buyer  makes  its  decisions  alone,  and  later  in  a  situation  where  members  of  a  two-level  supply 
chain  coordinate  their  actions. 

Introduction 

Numerous  probability  models  have  been  suggested  for  representing  uncertain 
demand  during  lead  time  (LT)  in  continuous-review  inventory  management  systems  when 
both  LT  and  demand  per  unit  time  (DPUT)  are  variable.  A  common  approach  to  finding  a 
distribution  for  lead  time  demand  (LTD)  involves  modeling  LT  and  DPUT  with  standard 
probability  density  functions  (PDFs).  Based  on  the  distributions  assigned,  a  compound 
probability  distribution  is  determined  for  demand  during  lead  time,  or  LTD.  The  latter 
distribution  is  used  to  determine  reorder  point  and  safety  stock  policies,  and  may  be  used  to 
estimate  inventory  costs.  In  some  cases,  analytical  formulas  for  optimal  reorder  point,  safety 
stock,  or  stockout  costs  are  available  in  terms  of  the  compound  distribution’s  parameters, 
while  in  other  situations  the  values  associated  with  certain  percentiles  of  the  compound  LTD 
distribution  are  estimated  to  provide  these  values.  Although  the  problem  of  finding  an 
appropriate  LTD  distribution  has  been  well  studied,  papers  written  in  recent  years  have 
continued  to  pursue  methods  that  overcome  unrealistic  distributional  assumptions  (Ruiz- 
Torres  &  Mahmoodi,  2010;  Vernimmen,  Dullaert,  Willime,  &  Witlox,  2008). 

This  paper  illustrates  an  approach  for  constructing  a  mixture  distribution  for  LTD  that 
allows  the  LT  and  DPUT  distributions  to  be  state-dependent.  This  method  also  allows  input 
distributions  that  take  any  standard  or  empirical  form.  Use  of  the  mixture  distribution 
technique  is  first  demonstrated  in  the  context  described  by  Cobb  (2013),  which  is  a  single¬ 
item  continuous-review  inventory  model  for  one  buyer.  For  single-firm  operating  in  a 
continuous-review  inventory  system,  the  mixture  distribution  method  for  modeling  the  LTD 
distribution  differs  from  the  typical  “moment-matching”  approach.  The  method  focuses  on 
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building  up  an  accurate,  closed-form  approximation  to  the  LTD  distribution  from  its 
components  by  using  mixtures  of  truncated  exponential  (MTE)  functions. 

After  the  mixture  distribution  approach  is  described,  a  two-level  supply  chain  model 
where  the  buyer  operates  under  uncertain  demand  and  utilizes  a  continuous  review 
inventory  system  will  be  considered.  In  this  two-echelon  supply  chain  model,  credit  terms 
(Chaharsooghi  &  Heydari,  2010),  quantity  discounts  (Li  &  Liu,  2006;  Chaharsooghi,  Heydari, 
&  Kamalabadi,  2011),  and  rebates  (Cobb  &  Johnson,  2013)  have  been  suggested  as 
coordinating  incentives  that  allow  the  supply  chain  members  to  divide  the  cost  savings 
resulting  from  coordinating  their  order  quantity  and  reorder  point  decisions.  In  each  of  these 
cases,  LTD  is  assumed  to  be  normally  distributed.  This  assumption  is  not  always  realistic, 
particularly  when  DPUT  and  LT  are  each  random  variables  such  that  LTD  has  a  compound 
probability  distribution  (Eppen  &  Martin,  1988;  Lau  &  Lau,  2003;  Lin,  2008).  This  paper  will 
incorporate  the  previously  described  model  (Cobb,  2013)  into  the  two-echelon  supply  chain 
problem  to  show  that  this  model  can  obviate  the  need  to  assume  that  demand  for  the  entire 
LT  period  is  normally  distributed. 

The  next  section  describes  LTD  distributions  and  uses  an  example  dataset  to  show 
how  standard  PDFs  can  be  used  as  approximations  to  the  LTD  distribution.  The  mixture 
distribution  method  is  also  used  for  the  example  problem.  Next,  the  different  approximations 
to  the  LTD  distribution  are  used  to  find  optimal  inventory  order  quantity  and  reorder  point 
policies.  This  is  followed  by  an  illustration  of  how  the  mixture  distribution  approach  can  allow 
more  complicated  LTD  distributions  to  be  incorporated  into  such  problems.  The  two-level 
supply  chain  model  is  then  introduced,  and  the  mixture  distribution  approach  is  used  to 
model  LTD  in  the  context  of  decentralized,  centralized,  and  coordinated  supply  chains.  The 
final  section  concludes  the  paper. 

Lead  Time  Demand  Distributions 

LTD  in  a  continuous-review  inventory  system  is  often  assumed  to  follow  a  compound 
probability  distribution.  Suppose  L  is  a  random  variable  for  lead  time  (LT)  and  D  represents 
random  demand  per  unit  of  time  (DPUT).  LTD  is  a  random  variable  X  determined  as 

X  —  Di  +  D2  +  ••■  +  /)[  +  •••  +  Dl  .  ( 1 ) 

Therefore,  Xis  a  sum  of  random,  independent,  and  identically  distributed  (i.i.d.) 
instances  of  demand.  The  mean  and  variance  of  X  can  be  calculated  as 

E(X)  =  E(L )  ■  E(D)  and  Var(X )  =  E(L)  ■  Var{D )  +  [E(D)]2  ■  Var(L).  (2) 

Suppose  the  data  in  Table  1  is  available  to  estimate  an  LTD  distribution.  This  table 
contains  50  observations  of  daily  demand  for  an  inventory  item  and  10  observations  for  LT 
on  orders  of  the  same  item.  The  expected  value  of  daily  demand  is  E(D)= 2.88,  and  the 
variance  of  this  random  variable  is  Var(D)= 2.84.  LT  has  an  expected  value  and  variance  of 
E(L)= 5.3  and  Var(L)= 6.9,  respectively.  According  to  the  formulas  in  Equation  2,  the 
expected  value  and  variance  of  LTD  are  E(X)=  15.26  and  Var(X)= 72.3,  respectively. 

The  remainder  of  this  section  will  illustrate  three  possible  methods  for  approximating 
the  LTD  distribution  underlying  the  data  in  Table  1. 

Normal  Approximation 

The  service  level  is  defined  as  the  percentage  of  replenishment  order  cycles  where 
demand  during  LT  is  satisfied.  To  determine  the  reorder  point  ( R )  required  to  achieve  a 
desired  service  level,  a  typical  textbook  approach  is  to  assume  the  LTD  distribution  is 
normal  and  use  normal  distribution  tables  or  Excel  formulas.  For  example,  to  find  the  R 
needed  to  achieve  a  95%  service  level  for  the  LTD  distribution  with  expected  value  and 
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variance  described  in  Table  1,  the  Excel  formula  NORM.INV(0.95,15.26,72.3A0.5)  can  be 
used  to  find  R  =  29.25. 


Table  1.  Observations  for  Daily  Demand  and  Lead  Time 


Daily  demand  (DPUT) 
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1 
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Lead  time  (LT) 

3 

5 

3 

4 

4 

5 

5 

10 

5 

10 

The  normal  approximation  to  the  LTD  distribution  and  the  reorder  point  R= 29.25  are 
illustrated  graphically  in  Figure  1 .  By  implementing  this  policy,  we  would  expect  to  stockout 
on  5%  of  replenishment  order  cycles. 


504 

Distribution  for 

Demand  During  Lead  Time  (X) 

5.0J 

NormaK  15.26.72.3) 

3.02 

3.01  / 

ROP=29.25 

\ 

Area=0.05 

10  20  JO  40  SO  60 


Figure  1.  LTD  Distribution  and  Reorder  Point 
Negative  Binomial  Approximation 

Although  the  normal  approximation  to  the  LTD  distribution  is  popular,  there  are 
numerous  other  approximations  that  have  been  suggested  in  the  literature.  For  example, 

T aylor  (1961)  suggested  using  the  negative  binomial  (NB)  distribution  for  the  case  where  the 
Poisson  distribution  is  a  good  fit  for  DPUT  and  LT  has  a  gamma  distribution.  We  denote  the 
approximate  LTD  distribution  by  /.  Here  we  assume  the  NB(r,p)  distribution  for  LTD  is 

/(x;r,p)  =  ^^(1 -p)rp*  x  =  0,1,2, ...  (3) 

where  r(-)  is  the  gamma  function.  Given  this  formulation,  £(X)=rp/(1-p)  and  Var(X)=E(X)l(  1- 
p).  There  are  two  ways  of  finding  a  reorder  point  that  will  provide  an  appropriate  service 
level  with  this  NB  formulation.  Taylor  (1961)  provided  a  formula  to  calculate  stockout 
probabilities  as  a  function  of  the  underlying  Poisson  and  gamma  distributions.  These  can  be 
calculated  for  possible  reorder  point  values  until  a  suitable  value  that  meets  the  service  level 
objective  is  found.  Excel  can  also  be  used  to  enumerate  the  probabilities  of  achieving  a 
certain  service  level  with  various  possible  values  of  R.  Unfortunately,  the  built-in 
NEGBINOM.DIST  function  only  accepts  integer  values  of  the  r  parameter,  so  these 
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probabilities  must  be  calculated  using  the  formula  in  Equation  3  and  the  GAMMALN 
function. 


For  the  data  in  Table  1 ,  we  can  use  the  empirical  expected  value  and  variance  to 
solve  two  equations  and  two  unknowns  and  obtain  r  =  4.08  and  p  =  0.79.  This  NB 
distribution  is  shown  in  Figure  2.  The  value  of  R  that  provides  approximately  a  95%  service 
level  is  R  =  31 . 


Figure  2.  Negative  Binomial  Distribution  for  Lead  Time  Demand 


This  solution  is  essentially  the  same  as  the  one  found  using  Taylor’s  (1961) 
analytical  formulas.  In  this  case,  the  Poisson  daily  demand  assumption  may  be  reasonable 
because  E(D)  and  Var(D)  are  very  similar,  a  feature  of  the  Poisson  distribution. 

Mixtures  of  Truncated  Exponentials  Approximation 

The  functional  form  of  some  PDFs,  such  as  the  negative  binomial  PDF  in  Equation  3, 
does  not  permit  integration  in  closed-form.  The  means  that  the  result  of  an  expected  value 
calculation  with  such  a  PDF  does  not  have  a  functional  form  that  can  be  used  for  further 
computation.  These  calculations  could  include,  for  example,  building  a  cost  function  to 
perform  nonlinear  optimization  to  find  optimal  inventory  policies.  One  approach  suggested  to 
overcome  this  limitation  is  the  MTE  model  (Moral,  Rurm,  &  Salmeron,  2001). 

An  example  of  a  four-piece,  two-term  (ignoring  the  constant)  MTE  function  that  can 
be  used  to  model  LTD  given  an  LT  of  L  =  3  for  the  problem  in  the  previous  section  is 

/x|{L  =  3}(x)  — 

f- 0.7148  +  0.6681  exp{0.0325x]  +  0.000048  exp{0.989x} 

-96.721  -  318.54  exp{-1.945x}  +  96.76  exp{0.000128x} 

'  0.1383  -  (1.63 E  -  06)  exp{x}  +  (2.89 E  -  09)exp{1.5x] 

,-0.0252  +  0.9786  exp{-0.205x] 


if  2.5  <  x  <  5 
if  5  <  x  <  8 
if  8  <  x  <  11.5 
if  11.5  <  x  <  17.5 


(4) 


This  function  was  found  by  simulating  500  series  of  three  observations  for  daily 
demand  from  values  in  Table  1  using  a  bootstrapping  approach.  The  constants — coefficients 
on  the  exponential  terms  and  coefficients  on  the  variable  X — were  determined  by  fitting  a 
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function  to  the  simulated  histogram.  There  is  an  established  literature  on  fitting  MTE 
functions  to  historical  data;  in  this  case,  the  method  suggested  by  Moral  et  al.  (2002)  was 
utilized.  A  graphical  view  of  the  MTE  function  overlaid  on  the  simulated  histogram  is  shown 
in  Figure  3. 


Figure  3.  Mixtures  of  Truncated  Exponentials  Distribution  for  Lead  Time  Demand  Given 

a  Lead  Time  of  Three  Days 

Similar  functions  fx\{L=i)  can  be  constructed  for  the  other  possible  LT  values,  L  =  4, 

5,  and  10.  From  the  data  on  LT  observations  in  Table  1,  we  can  estimate  P(L= 3)  =  P(L= 4)  = 
P(L=10)  =  0.2  and  P{L= 5)  =  0.4.  A  mixture  distribution  approach  (Cobb,  2013)  can  be 
employed  to  find  the  LTD  distribution.  Here,  the  LTD  distribution  is  determined  as 

fxto  =  pa  =  3)  ■  fx  |{l=3}«  +  pa  =  4)  ■  fx  |{L=4}W  +  pa  =  5)  ■  ;*|{L=5} «  (5) 

+pa  =  10)  ■  fx\{L=io}a)- 

The  MTE  function  is  shown  in  Figure  4,  overlaid  on  the  previously  described  NB 
distribution.  This  MTE  function  has  17  pieces  and  up  to  six  terms  in  each  piece.  For 
illustrative  purposes,  a  continuous  NB  parameterization  is  displayed.  Because  the  class  of 
MTE  functions  is  closed  under  addition,  multiplication,  and  integration  (Moral  et  al.,  2001 ), 
the  mixture  distribution  resulting  from  the  calculation  above  is  also  an  MTE  function.  Thus,  it 
retains  the  same  desirable  mathematical  properties. 

We  can  perform  closed-form  integrations  of  the  MTE  LTD  distribution  to  find  a 
reorder  point  that  achieves  a  desired  service  level.  In  this  case, 

Iq33  fx(.x)  dx  «  0.95  ,  (6) 

so  we  can  set  R  =  33.3  to  obtain  a  95%  service  level. 
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Figure  4.  Mixtures  of  Truncated  Exponentials  Lead  Time  Demand  Distribution  Overlaid 

on  a  Negative  Binomial  Approximation 

The  next  section  discusses  the  use  of  the  MTE  function  for  finding  inventory  policies 
in  a  continuous-review  inventory  system. 

Finding  Inventory  Policies 

Suppose  that  we  want  to  determine  an  optimal  order  quantity  and  reorder  point  in  a 
continuous-review  inventory  system  (a  “( Q,R )”  policy).  We  consider  four  models  that  could 
be  used  to  find  the  best  policy  given  the  data  available  (see  Table  1):  (1)  a  normal 
approximation  to  the  LTD  distribution;  (2)  the  NB  approximation  to  the  LTD  distribution;  (3) 
the  MTE  mixture  distribution;  and  (4)  a  simulation-optimization  model  that  simulates  LT  and 
LTD  values  from  the  empirical  distributions  developed  from  Table  1 .  We  term  the  latter 
model  the  “actual”  solution. 

A  simple  cost  function  with  no  backordering  allowed  (Johnson  &  Montgomery,  1974) 
for  this  problem  is 

TC(Q,R)  =K-^  +  ?^  +  h-(0.SQ  +  R-E(X)).  (7) 

In  this  equation,  K  is  the  fixed  cost  per  order,  Y  is  the  expected  annual  demand,  h  is 
the  holding  cost  per  unit  per  year,  and  tt  is  the  stockout  cost  per  unit.  The  average  inventory 
includes  safety  stock  of  R-E(X).  The  shape  of  the  distribution  for  LTD  determines  the 
expected  shortage  per  cycle,  SR.  For  a  given  reorder  point, 

sr  =  J“0  -  R)  ■  fx O)  dx.  (8) 

Suppose  Y=E(D)  ■  250  working  days  =  720,  K=  30,  h=4,  and  tc=5.  The  key  to  finding 
an  optimal  ( Q,R )  combination  is  to  evaluate  SR  as  part  of  constructing  the  total  cost  function 
in  Equation  7.  With  the  MTE  function,  the  calculation  in  Equation  8  can  be  performed  in 
closed-form,  and  the  result  substituted  into  Equation  7  to  obtain  a  closed-form  total  cost 
function.  The  expected  shortage  per  cycle  as  a  function  of  R  is  an  eight-piece  expression, 
with  selected  terms  shown  below: 


m  khsukm 
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if  16.15  <  r  <  16.5 


SR(r)  = 

f- 3876.5  +  4.66  exp{-0.205r}  +  6.31  exp{-0.172r} 

+3888.1  exp{0.005r}  -  21.82r  -  0.04r2 
-3890.6  +  4.66  exp{-0.205r}  +  6.31  exp{-0.172r} 

^  +3888.1  exp{0.005r}  +  20.6  exp{-0.140r}  -  20.64r  -  0.07r2  if  16.5  <  r  <  17.5 
*  -3889.2  +  6.31  exp{-0.172r}  +  3888.1  exp{0.005r} 

+20.6  exp{— 0.140r}  -  20.76r  -  0.07r2  if  17.5  <  r  <  23.5 

,-8.74  +  29.87exp{— 0.78r)  +  0.28r  -  0.002r2  if  31  <  r  <  46.5  . 


(9) 

Optimization  over  the  resulting  cost  function  is  fast.  The  example  here  was  solved  in 
Mathematica  9.0  by  using  the  ArgMin  function.  The  results  obtained  using  the  four  methods 
under  consideration  are  shown  in  Table  2.  An  iterative  approach  (Hadley  &  Whitin,  1963)  in 
combination  with  numerical  integration  was  implemented  to  find  the  solutions  using  the 
normal  or  NB  approximations.  The  table  shows  the  values  Q*  and  R*  which — when 
implemented  simultaneously — minimize  annual  total  cost.  The  computing  (CPU)  times 
required  to  obtain  the  solutions  are  also  shown.  The  simulation-optimization  solution  was 
simply  stopped  after  running  for  several  hours,  and  the  values  obtained  were  assumed  to  be 
the  best  possible  solution. 


Table  2.  Results  for  Inventory  Policies  Determined  Using  Four  Approaches 


Method 

a* 

R* 

TC 

CPU  (sec.) 

Normal  Approximation 

108 

25 

482.99 

3.57 

NB  Approximation 

110 

25 

482.89 

3.76 

MTE  Mixture  Distribution 

110 

27 

481.10 

1.26 

Simulation-Optimization 

108 

27 

480.82 

oo 

Table  2  shows  that  the  MTE  mixture  distribution  works  equally  as  well  as  the  other 
approaches  when  implemented  to  obtain  an  optimal  ( Q,R )  policy.  The  next  section  illustrates 
that  the  mixture  distribution  approach  can  be  used  to  model  more  complicated  LTD 
distributions. 

State-Dependent  Variables 

The  advantage  of  the  mixture  distribution  approach  (Cobb,  2013)  in  inventory 
management  problems  is  that  more  complex  LTD  distributions  can  be  constructed  by 
building  the  model  from  its  components  while  still  maintaining  a  closed-form  representation. 
In  some  cases,  expert  knowledge  can  be  used  to  assign  state-dependent  distributions  for 
DPUT  and/or  LT. 

As  an  illustration,  suppose  the  first  row  of  10  observations  in  Table  1  can  be 
associated  with  replenishment  orders  where  a  significant  number  of  missions  were  canceled 
due  to  weather,  creating  reduced  demand.  This  reduced  demand  is  assumed  to  occur  on 
20%  of  replenishment  orders;  thus,  demand  can  be  considered  to  have  two  states:  regular 
(with  80%  probability)  and  low  (20%  of  the  time). 

To  demonstrate  another  approach  to  finding  MTE  approximations,  the  dataset  in 
Table  1  will  be  used  in  this  example  to  first  determine  a  standard  PDF  that  best  fits  the 
empirical  data  for  each  demand  state.  In  this  case,  the  log-normal  distribution  with  p  =  1.03 
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and  a 2  =  0.31  is  selected  for  the  regular  state,  and  the  LN(0.27,0.19)  is  chosen  for  state  2. 
The  demand  in  each  state  for  a  given  LT  period  is  then  a  sum  of  i.i.d.  log-normal  random 
variables.  This  sum  has  no  known  distribution,  but  approximations  for  the  PDF  of  a  sum  of 
log-normal  random  variables  exist.  Following  Cobb  et  al.  (2013),  the  Fenton-Wilkinson 
approximation  (Fenton,  1960)  is  implemented,  and  MTE  distributions  are  fit  to  these 
approximations  for  each  state  and  each  possible  LT  value.  For  state  1  and  state  2,  these 

functions  are  denoted  by  fx\{L=n  ar|d  fx\{L=iy  respectively.  The  conditional  PDF  for  LTD 
given  L  =  l  is  then  calculated  as 

/x|{L=i}(X)  —  0-8  ' /x|{l=i}(x)  +  0-2  '  /x|{L=Z}(X)-  (10) 

The  PDF  for  LTD  is  constructed  as  in  Equation  5.  The  new  LTD  distribution  is  bi- 
modal,  as  shown  in  Figure  5. 


Figure  5.  Mixture  Distribution  for  Lead  Time  Demand  With  State-Dependent  Demand 

Suppose  the  state-dependent,  bi-modal  distribution  shown  in  Figure  5  is  the  correct 
PDF  for  LTD.  Using  this  distribution  as  part  of  the  total  cost  function  to  find  the  optimal  ( Q,R ) 
policy  results  in  a  21%  savings  when  compared  to  implementing  the  policies  found  earlier 
using  the  MTE  distribution  shown  in  Figure  4  (or  one  of  the  other  approximations).  The 
mixture  distribution  approach  still  yields  a  closed-form  function  for  SR  and  the  optimization  is 
still  fast. 

Coordinated  Supply  Chains 

In  this  section,  we  consider  a  two-echelon  supply  chain,  as  depicted  in  Figure  6.  A 
buyer  experiencing  random  demand  places  its  orders  for  inventory  with  the  supplier. 


Figure  6.  Two-Echelon  Supply  Chain 

(Chaharsooghi  &  Heydari,  2010) 


rt»  soivtmjm 
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(11) 


The  cost  function  for  the  buyer  in  this  problem  is  as  follows: 

TCb{Q,  R,V)  =  (Kb-V)-^  +  ^+hs- (0.5  Q  +R-  E(X)). 

Most  of  the  notation  is  the  same  as  for  the  cost  function  defined  in  Equation  7.  The 
subscript  b  has  been  added  to  the  fixed  cost  per  order,  annual  unit  holding  cost,  and  total 
cost  to  identify  this  amount  with  the  buyer.  The  subscript  s  will  similarly  represent  the  seller. 
The  quantity  V\s  a  rebate  provided  by  the  seller  to  the  buyer  on  a  per  order  basis  as  an 
incentive  for  the  buyer  to  adopt  policies  that  benefit  both  parties  (Cobb  &  Johnson,  2013). 

As  discussed  in  the  introduction,  credit  options  and  price  discounts  have  also  been 
considered  in  this  two-level  supply  chain  as  coordination  incentives  (Chaharsooghi  & 
Heydari,  2010;  Chaharsooghi  etal.,  2011;  Li  &  Liu,  2006). 

The  cost  function  for  the  supplier  in  this  problem  is 

TCS(Q,  N,V)  =  (^+v)-^+  hs(N  -  1)0.5  Q.  (1 2) 

In  this  two-level  supply  chain  model,  the  buyer  selects  an  order  quantity  and  reorder 
point.  The  supplier  receives  orders  of  size  Q  from  the  buyer  and  purchases  inventory  from 
its  vendors  in  a  quantity  that  is  an  integer  multiple  N  of  the  buyer’s  order  size. 

The  supply  chain  can  operate  in  one  of  three  modes.  First,  the  buyer  can  select  Qd 
and  Rd  without  considering  the  effect  of  its  selection  on  the  supplier’s  costs.  In  response,  the 
supplier  selects  Ndto  minimize  its  own  costs.  This  is  referred  to  as  the  decentralized  mode, 
and  because  there  is  no  coordination,  the  rebate  amount  is  V  =  0.  Total  costs  in  the  supply 
chain  are  TCd=  TCb(Qd,Rd, 0)  +  TCs(Qd,Nd,0).  Second,  the  buyer  and  supplier  can  agree  on 
values  for  Qc,  Rc ,  and  Nc  that  minimize  the  sum  of  the  cost  functions  in  Equations  1 1  and  12. 
Because  the  members  cooperate  fully  and  are  centralized,  there  is  again  no  requirement  for 
the  supplier  to  provide  a  coordination  incentive  and  V  =  0.  Total  costs  in  this  mode  are 
denoted  by  TCC=  TCb{Qc,Rc, 0)  +  TCs(Qc,Nc,0). 

If  the  parties  are  not  centralized  but  can  coordinate  their  policies,  the  potential  exists 
to  divide  cost  savings  of  TC+  =  TCd -TCC.  An  interval  [Vmin,Vmax]  can  be  calculated  (Cobb  & 
Johnson,  2013)  such  that  any  value  for  the  rebate  V  in  the  interval  reduces  the  total  costs  in 
the  supply  chain  to  centralized  levels.  The  smallest  value  of  the  rebate  the  buyer  will  accept 
can  be  found  by  solving  TCb(Qc,Rc,V)  =  TCb(Qd,Rd, 0)  for  V.  This  value  is  denoted  by  Vmin. 

The  largest  value  of  the  rebate  the  seller  will  accept  can  be  found  by  solving  TCs(Qc,Nc,V) 
=TCs{Qd,Nd,0)  for  V.  This  value  is  denoted  by  Vmax.  For  the  example  in  this  paper,  we 
assume  that  if  the  parties  agree  to  coordinate  their  policies  (and  implement  Qc,  Rc,  and  Nc), 
the  value  of  the  rebate  they  select  is  V  =  (Vmin+Vmax)/2. 

All  of  the  two-echelon  supply  chain  models  referenced  previously  assume  that 
demand  for  the  entire  LT  period  is  normally  distributed.  For  the  case  where  both  Q  and  R 
are  selected  to  minimize  total  costs,  Charharsooghi  and  Heydari  (2010)  derived  expressions 
that  state  the  optimal  value  for  Q  (in  either  the  decentralized  or  centralized  mode)  as  a 
function  of  the  optimal  value  for  R  (and  vice  versa)  and  the  standard  normal  cumulative 
density  function.  The  optimal  values  can  be  found  by  iterating  between  these  two 
expressions.  The  supplier  selects  the  integer  value  for  N  that  minimizes  its  costs  subject  to 
the  choices  of  the  buyer. 

By  implementing  the  mixture  distribution  approach,  we  can  develop  closed-form 
expressions  for  the  cost  functions  in  Equations  1 1  and  12  and  find  optimal  solutions  in  the 
same  manner  as  the  solutions  presented  earlier  in  the  paper  for  the  ( Q,R )  inventory  model. 
For  illustration,  assume  Y  =  E(D)  ■  250  working  days  =  720,  Ks  =  Kb  =  30,  hs=  hb  =  4,  and  tt  = 
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5.  These  parameters  are  the  same  as  used  in  the  earlier  example  and  the  supplier  has  the 
same  cost  structure  as  the  buyer  (obviously,  this  may  not  always  be  true  in  practice). 

For  the  previous  example,  employing  the  MTE  mixture  distribution  in  Figure  4  gives 
the  same  results  in  Table  2  for  the  decentralized  case — Qd  =110  and  Rd  =  27.  In  this  mode, 
the  supplier  selects  the  multiple  of  the  buyer’s  order  quantity  that  minimizes  its  costs. 
Because  TCS(1 10,1,0)  =  197  and  TCS(1 10,2,0)  =  316,  the  supplier  selects  Nd=  1.  Total 
supply  chain  costs  in  the  decentralized  mode  are  TCd  =  678. 

In  the  centralized  mode,  we  find  the  optimal  order  quantity  and  reorder  point  that 
minimizes  TCb(Q,R,0)+TCs{Q,N,0)  for  several  possible  values  of  N,  then  choose  the  optimal 
values  that  give  the  lowest  combined  supply  chain  cost.  Again,  using  the  MTE  mixture 
distribution  allows  the  construction  of  a  closed-form  total  cost  function,  and  optimization  over 
this  function  in  Mathematica  is  fast.  Using  the  MTE  mixture  distribution,  we  find  that  Qc=154, 
Rc= 24,  and  A/c=1.  Total  supply  chain  costs  in  the  centralized  mode  are  TCd=  648.  Table  3 
summarizes  the  optimal  values  for  the  decision  variables  in  each  mode  and  the  total  costs 
for  each  party  and  the  supply  chain.  The  answers  obtained  with  the  mixture  distribution 
approach  are  compared  with  those  obtained  by  using  the  solutions  shown  by  Chaharsooghi 
and  Heydari  (2010). 

Table  3.  Optimal  Solutions  and  Total  Costs  for  the  Supply  Chain  in  Three  Modes 

of  Operation 


Normal 

Q 

R 

N 

\/ 

TCb 

TCS 

TC 

Decentralized 

108 

25 

1 

0 

483 

200 

683 

Centralized 

151 

23 

1 

0 

506 

143 

649 

Coordinated 

151 

23 

1 

8.53 

466 

183 

649 

MTE  Mixture 

Q 

R 

N 

\/ 

TCb 

TCS 

TC 

Decentralized 

110 

27 

1 

0 

481 

197 

678 

Centralized 

154 

24 

1 

0 

507 

141 

648 

Coordinated 

154 

24 

1 

8.51 

467 

181 

648 

A  comparison  of  the  solutions  in  the  decentralized  and  centralized  models  shows 
that  the  costs  in  the  entire  supply  chain  can  be  reduced  by  TC*  =  TCd -  TCC  =  30  if  the 
centralized  order  quantity  and  reorder  point  are  implemented.  However,  these  policies 
increase  costs  for  the  buyer  by  507  -  481  =  26.  By  using  the  solutions  in  Cobb  and  Johnson 
(2013)  to  find  the  value  V  that  divides  the  cost  savings  of  operating  in  the  centralized  mode 
between  the  buyer  and  the  seller,  the  buyer  is  adequately  compensated  for  increasing  its 
order  quantity.  The  rebate  amount  for  this  problem  is  8.51  per  order  cycle.  Both  members 
experience  costs  that  are  lower  than  in  the  decentralized  mode. 

Conclusions 

This  paper  serves  as  an  introduction  to  using  a  mixture  distribution  approach  to 
modeling  the  probability  density  function  for  LTD  in  problems  where  a  continuous  review 
inventory  system  is  implemented.  First,  construction  of  the  lead  time  distribution  was 
illustrated.  This  distribution  was  then  utilized  to  determine  optimal  order  policies  in  cases 
where  a  buyer  makes  its  decisions  alone,  and  then  when  members  of  a  two-level  supply 
chain  coordinate  their  actions. 
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This  paper  represents  the  first  stage  of  the  research  to  be  conducted  for  the  project 
entitled  “Modeling  Uncertainty  in  Military  Supply  Chain  Management  Decisions,”  which  has 
been  funded  under  BAA  Number  NPS-BAA-1 2-002  through  the  Naval  Postgraduate  School 
(Grant  N00244-1 3-1  -0014).  For  the  expanded  project,  inventory  requisition  data  for  a  five- 
year  period  has  been  obtained  from  the  Air  Force  Standard  Base  Supply  System  for  a 
power  supply  unit  used  on  F-15  and  F-16  aircraft.  The  techniques  presented  in  this  paper 
will  be  compared  to  an  approach  currently  used  by  the  Air  Force  that  employs  a  negative 
binomial  approximation  to  the  lead  time  demand  distribution.  The  comparison  will  be  similar, 
but  the  hypothetical  data  in  this  paper  will  be  replaced  by  the  actual  historical  data  provided 
by  the  Air  Force. 
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Abstract 

As  many  large-scale  DoD  systems  age,  and  due  to  budgetary  and  performance  efficiency 
concerns,  there  is  a  need  to  improve  the  decision  making  process  for  system  sustainment, 
including  maintenance,  repair,  and  overhaul  (MRO)  operations  and  the  acquisition  of  MRO 
parts.  To  help  address  the  link  between  sustainment  policies  and  acquisition,  this  work 
develops  a  greedy  heuristic-based  local  search  algorithm  to  provide  a  system  maintenance 
schedule  for  multi-component  systems,  coordinating  recommended  component  maintenance 
times  to  reduce  system  downtime  costs  thereby  enabling  effective  acquisition. 

Introduction 

Large  organizations  such  as  the  Department  of  Defense  (DoD)  have  to  devote  a 
significant  amount  of  their  budgets  to  system  maintenance.  According  to  a  2007 
Government  Accountability  Office  (GAO)  report,  the  DoD  spends  approximately  40%  of  its 
budget  on  operations  and  management  (O&M)  activities  to  ensure  system  readiness 
($209.5  billion  in  2005).  GAO  reported  that  since  fiscal  year  2001,  the  DoD’s  O&M  costs  are 
increasing,  and  the  Air  Force,  in  particular,  had  to  increase  its  O&M  cost  by  29%.  As  many 
large-scale  DoD  systems  age,  and  due  to  budgetary  and  performance  efficiency  concerns, 
there  is  a  need  to  improve  the  decision  making  process  for  system  sustainment,  including 
maintenance,  repair,  and  overhaul  (MRO)  operations  and  the  acquisition  of  MRO  parts. 

The  DoD’s  acquisition  costs  have  seen  growth  in  recent  years  (GAO,  2013).  The 
GAO  (2013)  recommended  that  the  DoD  improve  its  strategic  management  plan  to  make 
maintenance  supply  chain  operations  more  cost  effective.  Further,  the  DoD  was  advised  to 
“link  acquisition  and  sustainment  policies”  for  depot  maintenance  improvement  and  ultimate 
cost  efficiency  (GAO,  2011).  To  help  address  the  link  between  sustainment  policies  and 
acquisition,  this  work  develops  a  framework  to  provide  a  system  maintenance  schedule  for 
multi-component  systems.  As  the  multiple  components  of  a  system  have  their  own 
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lifecycles,  an  efficient  means  to  schedule  overall  system  maintenance  should  consider  these 
individual  components  to  maximize  long-term  availability  of  the  system.  This  framework 
coordinates  recommended  maintenance  times,  such  as  those  found  as  a  result  of  reliability 
centered  maintenance  (RCM)  or  from  original  equipment  manufacturer  (OEM)  suggestions, 
to  formulate  a  system-level  maintenance  schedule  for  a  finite  time  horizon.  Such  a 
framework  will  increase  the  acquisition  efficiency  of  components  with  a  more  effective 
system-level  maintenance  schedule. 

With  the  recent  computational  advances,  several  preventive  maintenance  models 
have  been  proposed  for  complex  multi-component  systems  considering  component 
interactions.  In  the  preventive  maintenance  scheduling  problem  (PMSP),  different  kinds  of 
component  interactions  are  taken  into  account.  Interaction  among  components  can  be 
economic  dependence,  structural  dependence,  and/or  stochastic  dependence  (Thomas, 
1986).  In  a  basic  sense,  economic  dependence  among  system  components  means  that  the 
cost  of  joint  repair  is  different  from  cost  of  individual  repair  (Dekker,  Wildeman,  &  van  der 
Duyn  Schouten,  1997),  suggesting  that  performing  repair  operations  for  multiple 
components  at  once  can  be  done  with  less  expense  than  for  single  components. 

Researchers  have  considered  different  model  formulations,  as  well  as  solution 
techniques,  to  address  preventive  maintenance  decision  making.  Stinson  and  Khumawala 
(1987)  formulated  a  heuristics-based  mixed  integer  linear  program  (MILP)  model  for  a  finite 
horizon  preventive  maintenance  problem  for  maintaining  machines  in  series.  Budai, 
Huisman,  and  Dekker  (2006)  proposed  a  heuristics-based  MILP  solution  for  scheduling 
railroad  network  maintenance.  Other  few  noteworthy  approaches  are  Bayesian  network 
model  (Celeux,  Corset,  Lannoy,  &  Ricard,  2006),  goal  programming  for  a  multi-objective 
problem  (Bertolini  &  Bevilacqua,  2006),  and  dynamic  programming  (Dekker,  Wildeman,  & 
Van  Egmond,  1996). 

In  terms  of  algorithm  development,  Dekker,  Smit,  and  Losekoot  (1991)  presented  an 
optimal  maintenance  model  using  a  set-partitioning  algorithm  for  multiple  maintenance 
activities.  One  downside  of  their  model  was  that  they  considered  each  activity  time  to  be 
negligible  relative  to  the  total  planning  horizon.  Later  Dekker  et  al.  (1996)  solved  the  above- 
mentioned  problem  with  a  dynamic  programming  formulation,  concluding  that  the  dynamic 
algorithm  is  a  good  heuristic  for  rolling  horizon-based  problems  which  can  incorporate 
short-term  system  information  for  decision  support.  Dekker  et  al.  (1997)  provided  a  review  of 
maintenance  models  for  multi-component  systems,  which  covered  economically  dependent 
systems.  The  Markov  decision  chain-based  approach  was  also  studied  by  Dekker  et  al. 
(1996)  for  the  multi-activities  maintenance  problem  which  was  applicable  to  systems 
consisting  of  many  components.  Previous  Markov  chain-based  models  were  limited  to  few 
components.  An  opportunistic  maintenance  policy  was  modeled  by  Gurler  and  Kaya  (2002) 
and  van  der  Duyn  Schouten  and  Vanneste  (1993)  for  identical  multi-component  systems. 
Sheu  et  al.  (1996)  modeled  a  similar  kind  of  problem  with  a  two-stage  opportunistic  policy.  In 
the  case  of  non-identical  components  maintenance,  the  tradeoff  between  the  repair  cost  of 
one  component  versus  another  should  be  considered,  including  the  resulting  increase  in  the 
complexity  of  the  model. 

PMSP  remains  a  very  active  area  of  research.  Little  work  in  this  field  has  used 
heuristics  and  meta-heuristics  based  methodologies  to  model  preventive  maintenance 
framework  (Nicolai  &  Dekker,  2008).  A  new  meta-heuristic  based  on  a  genetic  algorithm  was 
applied  in  train  maintenance  scheduling  problems  by  Sriskandarajah,  Jardine,  and  Chan 
(1998),  primarily  optimizing  cost.  Nicolai  and  Dekker  (2008)  presented  a  review  of 
preventive  maintenance  and  recommended  that  more  researches  need  to  be  done  in  this 
area  developing  more  heuristic  and  meta-heuristic  approaches  like  simulated  annealing  and 
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local  search.  Meta-heuristic  based  algorithms  have  proven  very  successful  for  flowshop 
scheduling  problems  (Pan  &  Ruiz,  2012),  which  have  similar  characteristics  to  preventive 
maintenance  scheduling. 

This  work  presents  a  greedy  heuristic-based  local  search  algorithm  for  preventive 
maintenance  of  multiple  components  which  would  be  a  new  contribution  in  this  field  of 
research.  We  develop  a  local  search-based  algorithm  to  minimize  the  total  maintenance 
cost  of  a  system  over  a  finite  planning  horizon.  This  paper  is  organized  as  follows.  The 
Methodological  Development  section  provides  a  detailed  description  of  the  different 
components  and  procedure  of  our  proposed  schedule  algorithm  for  a  multi-component 
system.  The  next  section,  Greedy  Heuristic  with  Local  Search  Algorithm,  provides 
experimental  results  for  a  presented  multi-component  scheduling  problem.  We  conclude  our 
paper  with  the  Experimental  Results  section  and  some  concluding  remarks. 

Methodological  Development 

Here  we  develop  a  new  formulation  and  solution  algorithm  to  address  preventive 
maintenance  scheduling  for  a  multi-component  system.  It  is  assumed  that  maintenance 
results  in  a  “good  as  new”  condition. 

Baseline  individual  component  maintenance  times  for  planning  horizon  T  (i.e., 
system-in-use  time)  are  known  and  recommended  based  on  a  mean  time  between  failure 
(MTBF)  calculation  (e.g.,  by  RCM  or  OEM  calculations).  We  assume  these  component 
maintenance  times  are  given  in  their  in-use-time  or  up-time.  Our  goal  is  to  suggest  to  alter 
the  recommended  maintenance  schedule  for  a  multi-component  system  in  a  joint  manner  for 
as  many  components  as  possible.  Performing  many  individual  maintenance  events  at 
recommended  schedules  can  potentially  lead  to  cost  savings  due  to  reduced  setup  costs 
and  reduced  downtime.  However,  varying  too  far  from  recommended  MTBF  guidance  can 
lead  to  unnecessary  maintenance  (in  the  earliness  situation)  and  risk  of  failure  (in  the 
tardiness  situation).  Earliness  refers  to  the  performance  of  maintenance  earlier  than 
recommended,  with  tardiness  representing  the  performance  of  maintenance  at  a  time  later 
than  recommended.  As  such,  there  are  penalties  associated  with  both  earliness  and 
lateness,  as  well  as  a  penalty  for  system  downtime  while  maintenance  is  being  performed. 

Different  potential  maintenance  schedules  can  be  compared  and  evaluated  using  a 
penalty  function  approach  (Yousefi  &  Yosuff,  2013).  In  this  approach,  a  penalty  function  can 
be  achieved  by  quantifying  setup-related  costs  into  setup  penalties,  downtime  costs  into 
downtime  penalties,  related  expense  (i.e.,  costs  of  unnecessary  maintenance)  of  earliness 
into  earliness  penalties,  and  potential  failure  costs  of  tardiness  situation  into  tardiness 
penalties.  By  implementing  this  approach,  a  maintenance  schedule  can  be  found  which  will 
minimize  these  penalties.  These  penalties,  as  well  as  other  notation,  are  defined  as  follows: 
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T  Planning  horizon 
n  Number  of  components  in  the  system 
Cs  System  setup  penalty  per  maintenance 
i  Mi  maintenance  time  for  component  / 

Ce,i  Earliness  penalty  for  component  /,  per  unit  time 

Cl.i  Tardiness  penalty  for  component  l,  per  unit  time 

Cd  System  downtime  penalty  per  unit  time 
Tri  Component  maintenance  duration  for  component  / 
d  Construction  phase  time-span  parameter  where  5  e  (0,  1] 
y  Joint  maintenance  time  parameter  y  e  (0,  1] 

A j  Deviation  of  individual  component  maintenance  times  from /th  system  maintenance 

Tmax  Maximum  time-span  of  construction  phase 
T  c  Construction  phase  time-span 
T mi  Set  of  first  component  maintenance  time 

T m2  Set  of  second  component  maintenance  time 

nc  Candidate  solution 
nd  Discard  solution 
Sc  Candidate  combination  set 
Sd  Discard  set 
S  Algorithm  solution  vector 

Decision  variables  for  the  scheduling  formulation  include  the  following: 

fJ  /th  system  maintenance  time 

R  Total  number  of  system  maintenance  events  scheduled  in  planning 
xj  If  feature  earliness  is  present  in  component  l  for  maintenance  j  (xj 

yj  If  feature  tardiness  is  present  in  component  /  for  maintenance  j  (yj 

z]  If  component  /  should  be  repaired  at  time  tJm  (z/  =  1)  or  not  (z]  = 

Performing  joint  repair  has  the  potential  to  save  maintenance  cost  because  for  many 
multi-component  systems  it  is  possible  to  perform  component  maintenance  simultaneously. 

Thus  total  repair  time  for  joint  maintenance  depends  on  individual  instance  and  can  be 
predicted  from  previous  system  maintenance  data.  Considering  all  these  penalties,  our  goal 
is  to  develop  an  algorithm  that  will  schedule  system  maintenance  time  such  that  total 
penalties  of  system  maintenance  are  minimized  over  the  given  planning  horizon  T.  The 
basic  optimization  problem  is  conceptualized  in  Equation  1,  where  CSR  represents  total 
setup  penalties  for  planning  horizon  T,  and  c(  represents  penalties  associated  with  /th 
system  repair  of  component  /.  C{  includes  penalties  for  downtime,  earliness,  and  tardiness 
for  component  /  during  /'th  system  maintenance.  Decision  variable  z(  determines  whether 
component  I  will  be  repaired  at  /'th  system  maintenance. 


horizon  T 

=  1)  or  not  (x/  =  0) 
=  1)  or  not  ( yj  =  0) 
0) 
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(1) 


nCsRXXZz>i 

s.  t.  z{  G  {0,1} 

MRO  requirement  constraints 


Equation  2  presents  the  actual  objective  function  and  constraints  for  the  problem  above. 


min  Cs  R  + 

ti 


zx 


\Tk  _  J  I 
\‘m,l  Lm\ 
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Zn  ^ — i  R 
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Ce,i  xi 

~  *4)  Cl, i  xi  + 


Zn  %  i  R 

,  l^oyTrA 


s.t.  R  >  0 

t’m  >  0 
x{  G  {0,1} 

y!  e  {0,1} 

z/  G  {0,1} 

y  e  (0,1] 


(2) 


One  of  the  decision  variables  is  the  system-in-use  time  at  which  system  maintenance  should 
be  performed.  As  maintenance  scheduling  is  multistage  (e.g.,  maintenance  is  a  repeated 
event),  the  time  at  which  maintenance  is  scheduled  for  iteration  j  is  tJm  This  work  will  solve 
Equation  2  over  a  finite  time  horizon  for  several  MRO  stages,  finding  a  series  of  tJm  values  at 
which  maintenance  should  occur.  OEM-recommended  individual  component  maintenance 
times  are  denoted  by  T^  L.  Here  tJm  values  attempt  to  coordinate  the  downtime  of  several 
components  to  maximize  long-term  availability  of  the  system.  Equation  2  conceptualizes  an 
availability  cost  problem,  where  z(  determines  whether  component  /  should  be  repaired  at 
time  t]m  according  to  the  cost  function  which  penalizes  unavailability.  Equation  2  also 
attempts  to  improve  upon  7^  to  minimize  the  deviation  of  individual  component 
maintenance  times  from  system  maintenance  time,  found  in  the  neighborhood  of  T^.  As 
such,  this  work  provides  the  maintenance  schedule  for  the  system,  whether  the  y'th 
maintenance  operation  will  repair  an  optimal  subset  of  the  n  components  in  the  system. 

Elements  of  the  above  formulation  are  given  more  detail  as  follows.  The  actual 
structure  of  the  penalty  function  here  can  vary  due  to  decision  maker  preferences. 

Penalty  Function 

Our  objective  is  to  minimize  total  system  maintenance  penalty  over  a  finite  time 
horizon  T.  Our  penalty  function  is  the  presented  objective  function  in  Equation  2.  This  total 
penalty  function  consists  of  system  setup  penalty,  system  downtime  time  penalty,  and 
penalty  for  any  deviation  of  individual  component  maintenance  times  from  system 
maintenance  time.  Note  that  we  are  not  penalizing  for  the  cost  of  performing  actual  repair, 
including  the  cost  of  acquisition  and  the  cost  of  labor,  among  others,  under  the  assumption 
that  this  cost  is  the  same  for  individual  repair  and  joint  repair. 
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System  Setup  Penalty 

The  setup  penalty  component  in  Equation  3  accounts  for  the  time  to  arrange  for 
system  maintenance.  A  system  setup  penalty  penalizes  all  associated  costs  for 
maintenance  setup,  charged  only  once  regardless  of  the  number  of  multiple  components 
involved  in  a  maintenance  work.  Not  included  is  component  setup  time,  as  that  is  not 
expected  to  be  a  factor  in  determining  individual  or  joint  maintenance;  any  maintenance 
performed  on  a  component  would  require  component  setup  time.  Fixed  system  penalty  per 
maintenance  work  Cs  is  known. 

System  setup  penalty  =  Cs  R 


Earliness  Penalty 

There  is  a  penalty  for  executing  the  component  maintenance  at  a  time  other  than  the 
maintenance  recommended  by  the  OEM.  If  system  maintenance  is  scheduled  earlier  than 
recommended  individual  component  maintenance,  then  there  is  a  penalty  for  early 
maintenance  work  for  that  component.  This  penalty  attempts  to  penalize  the  performance  of 
maintenance  unnecessarily  too  far  in  advance  of  the  OEM  recommendation,  and  it  is  a 
function  of  (i)  the  total  amount  of  earliness  determined  by  |t£;  -  tJm\  (ii)  the  earliness 
penalty  CEh  and  (iii)  whether  component  /  maintenance  is  performed  early,  determined  by 

*/■ 

Zn  v  i  R 

y "  |  t^i  —t]m\cEiix{ 


Tardiness  Penalty  Cost 

If  system  maintenance  is  scheduled  later  than  individual  component  maintenance, 
then  there  is  a  penalty  for  late  maintenance  work  for  that  component.  This  penalty  is  a 
function  of  the  deviation  of  recommended  individual  component  maintenance  times  from  the 
actual  system  maintenance  time.  The  penalty  is  higher  for  tardiness  than  earliness  here  due 

to  aversion  to  performing  maintenance  later  than  recommended.  This  is  represented,  in  part, 

2 

by  the  square  on  the  amount  of  tardiness  time,  (7^  ,  -  tJm)  .  Other  elements  include 
tardiness  penalty  CL  L  and  whether  component  /  maintenance  is  performed  after  the  OEM 
suggested  maintenance  time,  determined  by  y( . 

Zn  \  2 

/  CL,iy’l 

1=1  *—>j 


System  Downtime  Cost 

There  is  a  cost  associated  with  system  downtime  due  to  an  unproductive  or  idle 
system.  The  system  downtime  penalty  per  unit  time  CD  is  known.  Expected  component 
maintenance  duration  for  component  /  is  parameterized  as  Tri.  Parameter  y  represents  the 
percentage  of  total  expected  component  maintenance  duration  (i.e. ,  YJr,i  for  all  /  that  are 
present  in  y'th  system  maintenance)  that  would  be  the  expected  joint  maintenance  duration 
for  y'th  system  repair.  We  assume  this  y  value  to  be  constant  for  all  iterations.  The  value  of 
joint  maintenance  time  parameter  y  can  be  chosen  from  the  historical  data  of  a  related 
system  such  that  ye(0, 1],  The  higher  the  y  parameter  value,  the  higher  the  downtime 
maintenance  cost  would  be.  Higher  y  means  less  time  savings  in  joint  repair  compared  to 
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separate  maintenance,  y  reaches  a  value  of  1  when  the  expected  joint  repair  time  is  equal  to 
the  summation  of  individual  component  repair  times;  those  are  present  in  y'th  joint  repair.  In 
other  words,  the  expected  downtimes  are  the  same  for  joint  repair  and  separate  repair  when 

7  =  1. 

This  value  defines  joint  maintenance  times  for  a  multi-component  system.  The  term 
z[  determines  whether  the  y'th  maintenance  operation  for  component  /  is  performed. 

Zn  *  i  R 

y  CDy  Tri zj 


Construction  Phase  Time-Span  Parameter  (S) 

At  the  beginning,  construction  phase  time-span  parameter  delta  ( S )  is  chosen  such 
that  S  e  (0, 1],  This  S  value  is  kept  constant  throughout  the  algorithm.  Discussed  later,  the 
algorithm  solution  is  very  sensitive  to  this  delta  value  and  needs  to  be  tuned  according  to 
individual  instance.  A  detailed  sensitivity  analysis  and  tuning  recommendation  of  S  are 
presented  later. 

Weibull  Distribution 

The  recommended  individual  maintenance  times  are  assumed  here  to  be  the  MTBF 
from  a  two-parameter  Weibull  distribution.  The  Weibull  distribution  is  well  known  in  reliability 
analysis  in  describing  the  time  between  failures  for  components.  MTBF  for  a  Weibull 
distribution  is  found  in  Equation  7,  where  /?  is  the  shape  parameter,  77  is  the  scale  parameter, 
and  T  is  the  gamma  function. 

1 

MTBF  =  ^(l  +  -) 


Greedy  Heuristic  With  Local  Search  Algorithm 

The  maintenance  optimization  model  described  previously  is  solved  with  a  proposed 
iterative  Greedy  Heuristic  with  Local  Search  Algorithm  (GHLSA).  The  proposed  algorithm  is 
similar  to  the  generic  structure  of  the  Greedy  Randomized  Adaptive  Search  Procedure 
(GRASP;  Feo  &  Resende,  1995],  In  contrast  to  the  two  phases  of  GRASP,  our  proposed 
algorithm  has  three  phases:  (1)  a  construction  phase,  (2)  an  improvement  phase,  and  (3)  a 
local  search  phase.  In  the  GRASP  algorithm,  the  initial  solution  is  constructed  using  a 
randomized  sampling  technique,  whereas  our  algorithm  uses  a  greedy  heuristic  to  construct 
an  initial  partial  solution.  We  also  use  an  additional  improvement  phase,  where  the  greedy 
heuristic-based  improvement  ends.  An  overview  of  the  proposed  algorithm  is  presented  in 
Figure  1 . 
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procedure  GHLSA  () 
begin 

1  <—  Inputlnstance  {  } ; 

for  GHLSA  stopping  criterion  not  satisfied  — > 
nf  <—  InitialPartialSolution  (1,8); 
it]  <-  GHBI  (nf); 

Uj  <—  LocalSearch  (Uj  ); 
UpdateSolution  (7r; ); 

endfor 

return  OptimalSolutionFound; 
end  GHLSA; 


Figure  1.  Pseudo-Code  Overview  of  the  Proposed  Greedy  Heuristic  With 

Local  Search  Algorithm  (GHLSA) 

In  brief,  the  three  phases  of  the  algorithm  achieve  the  following: 

1 .  The  construction  phase  determines  how  many  components  in  the  system 
should  be  initially  examined  to  include  in  system  maintenance  of  multiple 
components  and  an  initial  estimate  for  the  time  at  which  that  multi-component 
maintenance  operation  should  occur. 

2.  The  improvement  phase  improves  the  construction  phase  result  by  dividing 
the  set  of  multiple  components  into  two  sets  (a  candidate  set  and  a  discard 
set)  to  determine  whether  dividing  the  maintenance  operation  will  produce  a 
lower  penalty  than  the  construction  phase.  This  phase  iterates  by  removing  a 
component  out  of  the  candidate  set  one  at  a  time  and  placing  it  in  the  discard 
set  and  calculating  penalty  improvement. 

3.  The  local  search  phase  focuses  on  the  resulting  candidate  set  from  the 
improvement  phase  and  iterates  across  the  different  times  associated  with 
recommended  component  maintenance  to  balance  the  penalties  of  earliness 
and  tardiness  of  individual  components. 

These  three  phases  are  performed  at  each  iteration  j,  thereby  resulting  in  the  set  of 
components  involved  in  the  y'th  system  maintenance  operation  and  the  time  at  which  the  y'th 
system  maintenance  operation  should  be  performed.  The  algorithm  stopping  criterion  is  the 
pre-determined  planning  horizon  T.  Let  /  be  the  set  of  discrete  time  periods  where  each 
element  represents  recommended  (e.g.,  from  RCM  or  OEM  suggestions)  repair  times  of  a 
component  during  planning  horizon  T. 

The  final  solution  of  this  algorithm  is  essentially  an  R  *  1  vector  for  all  system 
maintenance  operations,  where  each  element  of  the  vector  represents  the  recommended  y'th 
system  maintenance.  The  result  of  each  iteration  j  is  referred  to  as  the  y'th  partial  solution  of 
the  over  final  solution.  Each  element  of  the  algorithm  solution  is  comprised  of  two  parts:  ny 

[0]  refers  to  the  set  of  repair  times  ... ,  T^nAn  J  of  components  to  be  performed  jointly  at 

the  y'th  system  maintenance  operation  (where  T^A  is  the  an  maintenance  operation  for 

component  An),  and  7r;[1]  refers  to  the  recommended  time  tJm  at  which  the  y'th  system 
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maintenance  is  to  be  performed.  For  example,  Uj  =  [  TCj  [0],  TTj  [l  |]  =  [{T“A,  T^b,  T^  c},  tjm] 

suggests  that  the  ath  maintenance  operation  of  component  A,  the  bth  maintenance  of 
component  B,  and  the  cth  maintenance  of  component  C  will  all  be  performed  jointly  at  time 
t]m,  the  time  chosen  for  the y'th  system  maintenance  operation  to  occur.  Thus  during  each 
iteration  of  this  algorithm,  it  finds  an  element  which  we  refer  to  as  a  partial  solution  for 
algorithm  solution  set.  At  each  iteration  y,  the  three  phases  of  the  algorithm  are  performed, 
each  of  which  is  explained  in  detail  subsequently.  Through  these  three  phases  of 
construction  and  improvement,  a  partial  solution  is  found,  and  this  partial  solution  is  then 
added  to  the  solution  set  to  update  the  algorithm  solution  for  the  scheduling  maintenance 
problem.  This  iterative  process  is  completed  when  the  solution  is  found  for  the  given 
planning  horizon. 

Using  input  instance  /  and  chosen  value  8,  an  initial  partial  solution  n°  is  created  in 
the  construction  phase.  During  the  improvement  phase,  this  initial  partial  solution  7r°  is 
improved  using  greedy  heuristic-based  procedure  GHBI.  This  improved  partial  solution  is 
represented  by  n During  the  local  search  phase  of  the  y'th  iteration,  partial  solution  7r;'.is 
further  improved  using  the  LocalSearch  procedure,  and  the  third  phase  returns  the  final 
partial  solution  n'j.  After  finding  the  best  partial  solution  nj  in  the  third  phase,  we  need  to 
update  the  existing  algorithm  solution  S  and  input  set  /.  This  partial  solution  "  is  then  added 
as  the  y'th  element  to  solution  vector  S,  to  update  the  algorithm  solution.  All  scheduled 
component  maintenance  times  7^  ,  at  iteration  j  are  removed  from  set  /  for  the  next  (J  + 
l)st  iteration,  and  the  rest  of  the  unscheduled  component  repair  times  of  set  /  are  updated 
according  to  their  earliness  or  tardiness  deviation  for  y'th  system  maintenance. 

Phase  1:  Initial  Partial  Solution  Construction 

At  each  iteration  y,  the  first  phase  is  a  construction  phase  where  the  initial  partial 
solution  is  generated.  General  pseudo-code  for  this  partial  solution  construction  phase  is 
presented  in  Figure  2.  Tmax  is  the  time  duration  which  expresses  the  maximum  time-span 
which  includes  all  the  component  repair  times  to  be  initially  considered  for  repair  during  jth 
system  maintenance.  The  construction  phase  time-span  is  selected  according  to  the  8 
value,  which  reduces  the  length  of  time  originally  considered  by  proportion  8.  All  component 
repair  times  7^  (during  time-span  Tc  are  included  in  the  joint  repair  component  set  for  the 
initial  partial  solution  n°  for  iteration  j.  This  constructs  the  first  part  of  the  initial  partial 
solution,  7r°[0j. 

Step  1.1.  Calculate  Tmax 

The  maximum  time-span  of  construction  phase  Tmax  needs  to  be  calculated.  This 
Tmax  value  represents  the  time  duration  between  the  recommended  time  for  the  earliest  first 
repair  of  all  components  and  the  recommended  time  for  the  earliest  second  repair.  Let  the 
sets  of  first  and  second  repair  times  of  each  component  out  of  all  unscheduled  maintenance 
times  be  I'm!  and  Tm2>  respectively.  The  minimum  value  of  set  T-ml  is  denoted  by 
EarliestFirstRepairTime,  and  the  minimum  value  of  set  Tm2  is  expressed  by 
EarliestSecondRepairTime  in  the  pseudo-code  in  Figure  2.  The  absolute  value  of  their 
difference  is  the  value  of  time-span  Tmax 

Step  1.2.  Calculate  Tc 

Construction  phase  time-span  Tccan  be  calculated  by  multiplying  the  value  of  the 
maximum  time-span  of  construction  phase  Tmax  by  8.  In  a  sense,  8  is  the  scope  of 
granularity.  A  small  value  of  8  suggests  a  tight  granularity  of  the  maintenance  option  set, 
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meaning  that  a  shorter  time  frame  will  be  considered  for  Tc  with  which  to  consider  multiple 
component  maintenance  options.  For  a  larger  value  of  S,  Tmax  approaches  Tmax  value.  And 
Tc  is  equal  to  Tmax  when  5=1. 

Step  1.3.  Partial  Solution  Component  Set 

Insert  all  recommended  component  maintenance  times  x  that  are  originally 
scheduled  during  construction  phase  time-span  Tc  into  the  joint  repair  component  set  ^[O] 
of  the  initial  partial  solution  nj .  If  there  are  nml  elements  in  set  Tml,  then  it  would  take  nml 
iterations  to  construct  the  initial  partial  solution  component  set. 

The  time  at  which  system  maintenance  is  performed  on  the  components  in  nf[0] 
constitutes  the  second  part  of  the  initial  partial  solution,  7r°[1],  which  can  be  chosen 
according  to  several  heuristics  including 

•  the  mid-point  of  time-span  Tc, 

•  a  component  repair  time  of  component  set  7r°[0]  where  the  deviation  A is 
minimized,  or 

•  the  earliest  component  repair  time  (i.e.,  the  minimum  value  of  component  set 

nf  [0]). 

In  our  implementation,  the  third  heuristic  above  is  used  to  construct  the  later  part  of 
the  initial  partial  solution.  That  is,  the  second  part  of  the  initial  partial  solution,  n°  [1],  is 
chosen  according  to  the  heuristic  convention  of  scheduling  system  repairs  at  the  earliest 
component  repair  time.  Thus,  this  phase  schedules  all  possible  component  maintenance 
during  time-span  Tc  at  the  earliest  possible  time  to  produce  an  initial  partial  solution. 

procedure  InitialPartialSolution  (1,8) 
begin 

rf  <-  {  }; 

Tmax  |EarliestFirstRepairTime  -  EarliestSecondRepairTime  |; 

^ml  <  ITnll 

T  <—  S  *  T 

1  c  ^  1  max 

for  i  <—  1  to  nml  do 

if  Tml[i\  <  Tc  then 

nf  <-nf  U  Tml[i]  ; 

endif 

endfor 

return  nf1; 

end  InitialPartialSolution; 


Figure  2.  Pseudo-Code  for  GHLSA  Phase  1,  the  Partial  Solution  Construction 

Phase 

Phase  2:  Greedy  Heuristic-Based  Improvement  (GHBI) 

During  the  second  phase  of  iteration  j,  the  algorithm  improves  the  initial  partial 
solution  nf  constructed  in  Phase  1,  focusing  primarily  on  the  components  in  7r°[0]  to  be 
repaired  jointly  (e.g.,  A  search  is  performed  in  the  neighborhood  of  n®  to 
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find  a  better  partial  solution.  This  combination  of  component  repair  times  is  improved 
according  to  a  greedy  heuristic  of  removing  the  last-one-out  (i.e. ,  latest  component  repair 
time)  from  existing  combinations. 

Let  the  initial  partial  solution  nf  be  the  existing  partial  solution  nj  (i.e.,  y'th  solution 
element).  If  there  are  np  elements  in  the  joint  repair  component  set  (7r°[0])  of  the  existing 
partial  solution,  then  there  would  be  np  possible  combinations  of  component  sets  that  can  be 
created  according  to  the  last-one-out  greedy  heuristic.  The  best  combination  set  among  np 
possible  combinations  is  selected  in  ( np  -  1)  iterations.  At  each  iteration  of  the  ( np  -  1),  two 
temporary  partial  solution  elements  called  candidate  solution  nc  and  discard  solution  nd 
(i.e.,  temporary y'th  and  (/+1)st)  are  generated  from  existing  partial  solution  n'-.  The  best 
candidate  solution  is  selected  as  the  new  existing  partial  solution  nj  according  to  an 
acceptance  criterion.  Each  iteration  of  this  greedy  heuristic-based  improvement  method, 
which  is  the  ImproveCombination  procedure  in  Figure  3,  is  described  below. 

Step  2.1.  Determining  nj[0] 

The  first  part  of  a  solution  element  presents  the  component  repair  times  to  be 
repaired  jointly.  Improved  combination  of  this  joint  repair  component  set  is  searched  using 
the  last-one-out  heuristic.  To  generate  an  improved  combination  of  the  y'th  solution  element, 
two  sets  (i.e.,  candidate  combination  set  Sc  and  discard  set  Sd)  are  created  from  the  existing 
joint  repair  component  set.  The  candidate  set  will  eventually  be  repaired  during  the  y'th 
iteration,  and  the  discard  set  will  be  saved  for  the  (/  +  1)st  iteration  or  beyond.  Let  the 
existing  joint  repair  component  set  be  the  initial  value  of  candidate  combination  set  Sc.  By 
applying  the  last-one-out  greedy  heuristic  (i.e.,  latest  component  repair  time),  a  new  discard 
set  Sd  is  created.  To  generate  the  discard  set  Sd,  the  latest  component  repair  time  (i.e.,  max 
Sc )  is  removed  from  candidate  solution  set  Sc  and  inserted  into  discard  set  Sd.  Candidate 
set  Sc  and  discard  set  Sd  construct  the  first  part  of  the  candidate  solution  nc  and  discard 
solution  nd  respectively  (i.e.,  nc[0]  and  7rd[0j). 

Step  2.2.  Determining  nj[l] 

The  time  at  which  the  elements  of  the  candidate  solution  7tc [0]  are  repaired  is  found 
from  the  earliest  component  repair  time  heuristic  for  the  set  (i.e.,  min  Sc).  This  time  of  repair 
is  7rc [1] .  Similarly,  the  components  in  discard  solution  7rd [0]  are  repaired  at  7rd[l],  or  min  Sd. 
Other  heuristics  that  could  be  used  in  this  step  were  presented  in  step  3  of  the  previous 
phase. 

Step  2.3.  Acceptance  Criterion 

The  candidate  solution  is  selected  as  the  existing  partial  solution  nj,  according  to  the 
acceptance  criterion  of  the  minimum  penalty  function.  The  existing  candidate  solution  is 
chosen  as  the  partial  solution  nj  if  the  combined  penalty  function  value  of  candidate  and 
discard  solutions  is  less  than  the  penalty  function  value  of  the  existing  partial  solution  nj. 

Figure  3  presents  the  procedure  of  developing  new  combination  set  according  to  the 
greedy  heuristic. 
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procedure  ImproveCombination  ( 7 r°) 
begin 

CurrentPenalty  <—  PenaltyFunction  (tc®); 

n'i  «-  nf ; 

<-  [  ] ; 

T*d  <-  [  ] ; 

5C  <-  7r/[  0  ]; 

Sd  <-  {  }; 
np  *—  \n]°[  o  ]l; 

for  i  <—  1  to  (np  -1)  do 

Sc  <—  remove  last  component  repair  time  and  insert  it  in  Sd  ; 

[{  Sc  },  min  (  Sc  )  ]  ; 
ltd  [{  Sd  },  min  (  Sd  )]  ; 

NewPenalty  <—  PenaltyFunction  (nc  )+  PenaltyFunction  (nd)\ 

if  NewPenalty  <  CurrentPenalty  then  %  Acceptance  criterion 

TTj  <■  7 Tc  , 

CurrentPenalty  <—  PenaltyFunction  (nc  )  ; 

endif 

endfor 
return  7 Zj  ; 

end  ImproveCombination  ; 


Figure  3.  Pseudo-Code  for  Improving  Combination  Stage 

As  long  as  the  number  of  elements  np  of  existing  partial  solution  n '  is  greater  than  1 
and  minimizes  the  penalty  function  value,  nj  is  divided  into  two  new  parts:  candidate 
solution  7rc  and  discard  solution  7rd.  This  iterative  improvement  is  performed  in  the  while 
loop  presented  in  procedure  GHBI.  Figure  4  describes  the  procedure  GHBI  using  pseudo¬ 
code. 
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procedure  GHBI  (n°) 


begin 


CurrentPenalty  <—  PenaltyFunction  ( n y°); 
np  <-  |7r/[0]|; 


NewPenalty  <—  0; 

while  (NewPenalty  <  CurrentPenalty  and  np  >  1  )  do 


CurrentPenalty  <—  PenaltyFunction  (  Uj  ); 


Tij  <—  ImproveCombination  ( 7i;  ); 


%  Using  greedy  heuristic  last-one- 


out 


NewPenalty  <—  PenaltyFunction  (  n y-  ); 


endwhile 
return  ny  ; 


end  GHBI; 


Figure  4.  Pseudo-Code  for  GHLSA  Phase  2,  the  Greedy  Heuristic-Based 

Improvement  Phase 


Phase  3:  Local  Search-Based  Improvement 

In  the  last  phase  of  system  maintenance  iteration  j,  an  improved  partial  solution  is 
selected  by  searching  the  neighborhood  of  current  partial  solution  nj,  building  the  best 
candidate  set  of  components  repair  at  the  y'th  iteration.  Let  this  improved  partial  solution  be 
nj'  and  its  initial  value  be  n-.  Emphasis  in  this  third  phase  is  placed  primarily  on  searching 

different  values  of  t]m  in  the  neighborhood  of  7r;'[l]  to  determine  when  the yth  maintenance 
operation  should  occur.  The  pseudo-code  for  this  local  search  phase  is  shown  in  Figure  5. 
During  this  improvement  phase,  tJm  iteratively  takes  the  values  of  component  maintenance 
time  generated  from  the  final  combination  set  nj  [0]  during  the  previous  phase  and  creates  a 
temporary  partial  solution.  During  this  iterative  process,  the  partial  solution  is  updated 
according  to  the  penalty  function  in  Equation  2.  According  to  our  selected  method,  it  takes 
np  iterations  to  search  the  neighborhood  of  7r;'[l],  if  the  number  of  elements  in  combination 
set  nj [0]  is  np.  At  each  iteration  of  np,  a  new  temporary  partial  solution  called  temp  is 
generated.  Steps  of  each  iteration  are  as  follows. 

Step  3. 1.  Determining  nj'  [0] 

The  joint  repair  component  set  comprising  7r"[0]  takes  the  value  of  the  final 
combination  set  (i.e.,  7ry'[0] )  found  in  the  second  phase. 

Step  3.2.  Determining  nj'[l] 

During  this  improvement  phase  temp[l]  (i.e.,  tJm)  iteratively  takes  the  values  of  the 
component  maintenance  time  generated  from  the  final  combination  set  nj  [0]  during  the 

previous  phase.  At  iteration  np,  tJm  would  take  the  value  of  npth  element  of  combination  set 
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Step  3.3.  Acceptance  Criterion 

The  acceptance  criterion  is  the  value  of  the  penalty  function  presented  in  Equation  2. 
At  each  iteration  of  np,  the  temporary  partial  solution  temp  is  selected  as  the  new  existing 
partial  solution  only  if  the  new  temporary  partial  solution  minimizes  the  penalty  function 
value. 


At  the  end  of  np  iterations,  the  LocalSearch  procedure  returns  the  best  value  found  in 
the  search.  The  return  value,  nj' ,  of  this  local  search-based  improvement  is  the  partial 
solution  representing  the  y'th  element  of  the  final  solution  vector. 


procedure  LocalSearch  (nj ) 
begin 

Uj  <—  Uj  , 

CurrentPenalty  <—  PenaltyFunction  (nj ); 

NoOfElement  <—  |7r;-[0]|; 
ifNoOfElement  >  1  then 

for  i  <—  1  to  NoOfElement  do 
temp  <—  Ttj  ; 
temp  [1]  <-  nj  [0]  [  i  ]; 

NewPenalty  <—  PenaltyFunction  (temp); 
if  NewPenalty  <  CurrentPenalty  then 
nj  [1]  n'j  [0]  [  i  ]; 

CurrentPenalty  =  PenaltyFunction  (nj ); 

endif; 

endfor; 

endif; 
return  Uj  ; 
end  LocalSearch; 

Figure  5.  Pseudo-Code  for  the  GHLSA  Phase  3,  the  Local  Search  Phase 
Experimental  Results 

An  example  problem  briefly  illustrates  the  algorithm. 

Problem  Specification 

Our  example  problem  addresses  10  components  in  a  multi-component  system.  We 
assume  the  initial  start  time  TNOW  is  zero.  We  assumed  the  earliness  penalty  and  tardiness 
penalty  values  to  be  equal  and  same  for  all  components  (i.e. ,  deviation  penalty  Cp). 
Maintenance  duration  Tr  l  is  assumed  to  be  5  time  units  for  all  components.  The 
recommended  individual  maintenance  times  of  these  components  are  assumed  here  to  be 
the  MTBF  from  a  two-parameter  Weibull  distribution  with  shape  parameters  (/?)  and  scale 
parameters  ( r )).  The  assumed  values  of  planning  horizon,  setup  cost,  downtime  cost  per  unit 
time,  earliness  penalty,  and  tardiness  penalty  are  presented  in  Table  1. 
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Original  Case 

The  original  case  follows  a  simple  procedure  for  maintenance.  Each  system 
maintenance  operation  is  performed  at  the  earliest  component  repair  time  (i.e.,  min  T^)  out 
of  unscheduled  component  maintenance  times.  It  is  assumed  that  all  repair  times  are  in  the 
system  repair  window  (i.e.,  min  [7^  ,  +  Tr  l])  and  will  be  scheduled  to  be  repaired  at  the  same 
time.  We  used  the  same  penalty  function  to  calculate  the  objective  function  value  for  the 
original  case.  Note  that  the  tardiness  penalty  will  always  be  zero  in  the  original  case 
instance,  as  system  maintenance  is  done  at  the  earliest  component  repair  time  and  there  is 
no  push  back  of  component  maintenance. 


Table  1.  Parameters  of  the  Illustrative  Example 


Component 

V 

P 

Other  values 

A 

15 

2 

TNOW  =0 

B 

20 

3 

T  =  200  time  unit 

C 

15 

3 

Cs  =30,000 

D 

17 

4 

CD  =5,000 

E 

23 

5 

Cp=CE,i=  Cl, 1=500,  for  all  / 

F 

37 

4 

Tr  i=  5500,  for  all  / 

G 

30 

7 

H 

22 

3 

I 

19 

2 

J 

26 

4 

Experimental  Evaluation 

We  solved  the  above-mentioned  problem  with  our  proposed  algorithm  (GHLSA)  and 
performed  a  comparative  study  between  the  original  case  results  and  GHLSA  results. 
Generated  experimental  penalty  function  data  were  transformed  into  percent  deviation  value 
(PD).  We  calculated  the  PD  of  objective  function  value  resulting  from  proposed  algorithm 
implementation,  from  the  original  case  result  using  the  following  equation,  where  Obj  original 
represents  penalty  function  value  for  original  case  and  Obj  ghlsa  represents  penalty  function 
value  produced  using  GHLSA  procedure.  Positive  PD  means  the  objective  function  value 
has  improved  (i.e.,  minimized)  using  the  proposed  algorithm  and  vice  versa. 


Percentage  Deviation  (PD) 


Obj  Original  Obj  GHLSA 
Obj  Original 


x  100 


All  calculated  results  for  given  instance  for  different  delta  values  are  presented  in 
Table  2. 

Sensitivity  Analysis  on  y 

Table  2  shows  that  for  a  given  instance,  the  proposed  algorithm  produced  a  very 
high  objective  function  value  which  resulted  in  negative  PD  value  for  lower  y  value  (i.e.,  y  = 
0.1  to  y  =  0.3).  For  y  value  greater  than  0.3,  calculated  PD  resulted  in  positive  values.  So  for 
higher  y  values  (i.e.,  for  y  >  0.3),  the  best  solutions  found  using  proposed  GHLSA  improved 
the  objective  function  value  of  the  original  case.  As  y  increased,  the  PD  value  decreased  for 
both  positive  and  negative  deviation  trends.  This  trend  was  true  for  any  y  value  (Figure  6). 
Collected  data  were  not  very  sensitive  to  y  value.  Trend  of  the  PD  remained  the  same,  and 
objective  function  value  changed  a  little  bit  with  a  change  in  y. 
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Sensitivity  Analysis  on  5 

For  all  y  values,  the  objective  function  percent  deviation  change  was  logarithmic  with 
8  (Figure  6).  For  lower  values  of  8,  GHLSA  produced  some  negative  deviation.  As  8 
increased,  it  generated  positive  deviation,  as  the  objective  function  value  decreased  with 
higher  8  value. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  565- 


Table  2.  Objective  Function  and  PD  Values  for  Given  Instance 


8 

y=0.1 

y=0.2 

II 

© 

II 

o 

4^ 

in 

© 

II 

y=0.6 

y=0.7 

00 

© 

II 

II 

O 

SO 

7=1 

Original 

1100101.52 

1355101.52 

1610101.52 
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Comparative  Study 

We  performed  a  comparative  study  of  the  original  case  and  the  GHLSA-based 
results  by  generating  different  instances  by  changing  given  value  of  Cs,  CD, and  CP. 

Sensitivity  Analysis  on  Setup  Penalty  Cs 

Produced  results  for  the  original  case  and  the  GHLSA  case  for  different  generated 
instances  for  six  different  setup  costs  are  presented  in  Table  3.  The  presented  values  are 
the  calculated  PD  values  for  the  best  objective  function  value  found  using  the  proposed 
GHLSA  for  each  instance. 


Figure  6.  Change  in  PD  Value  With  Delta 

For  all  generated  60  instances,  GHLSAs  were  able  to  improve  (i.e.,  positive  PD 
values)  the  original  case  penalty  function  value  (Table  3).  Improvement  ranged  from  1.08% 
to  20.56%  in  minimizing  the  objective  function  value  compared  to  the  original  case.  For  a 
given  Cs,  penalty  function  value  increased  as  y  decreased.  It  showed  an  increasing  trend  in 
PD  value  with  increasing  Cs,  for  any  given  y.  It  shows  the  potential  of  this  research  algorithm 
for  multi-component  system  maintenance  where  setup  cost  is  comparatively  high. 


Table  3.  PD  Values  for  Different  Setup  Costs 
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7.84 
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25k 
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8.83 

7.83 

7.03 

6.38 

5.84 

5.39 

20k 

14.84 

11.42 

9.28 

7.81 

6.75 

5.94 

5.30 

4.79 

4.37 

4.01 

15k 
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7.77 
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3.81 
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3.05 
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1.72 
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1.19 

1.08 

Sensitivity  on  Downtime  Penalty  CD 

We  generated  100  instances  for  10  different  CD  values  ranging  from  Ik  to  10k.  The 
calculated  PD  values  are  representative  of  the  best  solution  found  using  proposed  GHLSA 
at  granularity  level  0.1  (Table  4).  The  proposed  GHLSAs  were  able  to  improve  the  PD 
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values  of  all  100  instances  for  different  CD.  PD  values  ranged  from  3.80  to  25.24.  For  all  y, 
PD  value  decreased  with  higher  CD. 


Table  4.  PD  Values  for  Different  Downtime  Costs 


Downtime 

Cost 

y=0.1 

<N 

O 

II 

c*"* 

ll 

o 

7=0.4 

ll 

© 

Oi 

vo 

© 

II 

ll 

o 

Oi 

OC 

o 

II 

II 

O 

vo 

7=1 

lk 

25.24 

23.88 

22.66 

21.56 

20.56 

19.65 

18.82 

18.05 

17.34 

16.69 

2k 

23.88 

21.56 

19.65 

18.05 

16.69 

15.52 

14.51 

13.62 

12.83 

12.13 

3k 

22.66 

19.65 

17.34 

15.52 

14.05 

12.83 

11.80 

10.93 

10.18 

9.52 

4k 

21.56 

18.05 

15.52 

13.62 

12.13 

10.93 

9.95 

9.13 

8.44 

7.84 

5k 

20.56 

16.69 

14.05 

12.13 

10.67 

9.52 

8.60 

7.84 

7.20 

6.66 

6k 

19.65 

15.52 

12.83 

10.93 

9.52 

8.44 

7.57 

6.87 

6.28 

5.79 

7k 

18.82 

14.51 

11.80 

9.95 

8.60 

7.57 

6.76 

6.11 

5.57 

5.12 

8k 

18.05 

13.62 

10.93 

9.13 

7.84 

6.87 

6.11 

5.50 

5.01 

4.59 

9k 

17.34 

12.83 

10.18 

8.44 

7.20 

6.28 

5.57 

5.01 

4.55 

4.16 

10k 

16.69 

12.13 

9.52 

7.84 

6.66 

5.79 

5.12 

4.59 

4.16 

3.80 

Sensitivity  on  Deviation  Penalty  CP 

Different  CP  values,  ranging  from  100  to  1,000,  were  used  to  generate  100 
experimental  instances.  All  calculated  PD  values  of  the  original  case  and  the  GHLSA  case 
are  in  Table  5. 


Table  5.  PD  Values  for  Different  Deviation  Penalty  Values 
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12.98 
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In  all  experimental  instances  for  deviation  penalty  CP,  our  presented  GHLSAs  were 
successfully  able  to  minimize  the  penalty  function  value.  For  all  100  instances,  the  found  PD 
values  were  positive,  which  means  improvement  of  the  objective  function  value  compared  to 
the  original  case  study.  For  different  CP  values,  the  resulting  PD  values  ranged  from  4.37  to 
27. 85.  PD  value  decreased  with  higher  CP  for  all  y  (see  Table  5). 

Optimal  8  Value 

Tables  6  and  7  present  the  optimal  5  values  at  granularity  level  0.1 ,  for  generated 
instances  for  Cs  and  CP.  Note  that  optimal  8  values  for  downtime  instances  were  not 
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reported  in  the  tables.  For  all  100  instances  for  CD,  the  generated  optimal  delta  value  was 
0.8-1 .0  at  granularity  level  0.1 .  For  setup  cost,  all  instances  for  cost  ranging  from  15k  to 
30k,  optimal  8  was  0.8-1 .0.  For  10k  setup  cost  instances,  the  optimal  delta  value  was  1.0, 
and  it  decreased  to  0.5  for  the  lowest  setup  cost  5k. 


Table  6.  Optimal  8  Values  for  Different  Setup  Cost  Instances 
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Table  7.  Optimal  S  Values  for  Different  Deviation  Penalty  Cost  Instances 


DeviationPenalty 
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0.7 
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0.7 

0.7 

0.7 

1000 

0. 8-1.0 
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0. 8-1.0 
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0. 8-1.0 

0. 8-1.0 
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For  deviation  penalty  instances,  this  optimal  8  value  was  constant  for  all  values  of 
CP,  except  900.  For  CP  =  900,  for  lower  y  values  (i.e.,  y  =  0.1-0. 2),  the  8  value  for  best  found 
results  was  0.7-1 .0  and  for  the  rest  of  y  values,  it  was  0.7.  According  to  the  PD  analysis,  8 
is  a  significant  factor  in  finding  a  good  solution  by  implementing  this  greedy  heuristic-based 
methodology.  The  experimental  results  of  the  presented  260  instances  shows  that  a  higher 
value  of  8,  at  granularity  level  0.1,  is  a  safer  choice  when  the  penalty  function  for  all  the  8 
values  cannot  be  evaluated.  In  those  cases,  our  recommended  8  value  would  be  0.5-1 .0,  at 
granularity  level  0.1 . 

Tuning  Parameter  8 

The  presented  results  were  very  sensitive  to  8.  Tuning  of  this  granularity  parameter 
depends  on  the  system  configuration  (i.e.,  the  number  of  components)  and  available 
computation  power.  If  possible,  the  initial  tuning  can  be  done  at  granularity  level  0.1. 
Granularity  level  0.1  means  to  change  the  scope  of  granularity  8  value  by  0.1 .  With  a 
granularity  level  of  0.1,  in  10  runs  the  algorithm  would  generate  the  best  solution  possible 
with  the  proposed  method.  If  the  granularity  level  needs  to  be  smoother,  that  depends  on 
the  input  instance  (i.e.,  the  input  values  of  T^i).  If  7^ ;  values  result  in  a  very  small 
Tmax,  then  a  higher  granularity  level  may  not  produce  any  better  result,  as  a  number  of 
components  repair  times  may  remain  the  same  for  resulting  construction  phase  time-span 
T 

1  C ■ 

Concluding  Remarks  and  Future  Work 

In  this  paper,  we  proposed  a  greedy  heuristic  local  search  algorithm  for  multi- 
component  preventive  maintenance  scheduling  problems.  This  scheduling  algorithm  is 
based  on  some  greedy  heuristics  and  a  local  search  method.  This  new  algorithm  has  proven 
to  make  significant  improvement  of  the  objective  function  criterion,  compared  to  presented 
original  case  results.  We  have  implemented  the  presented  GHLSA  for  260  generated 
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instances  and  found  remarkable  results.  Deviation  analysis  showed  significant  improvement 
of  the  objective  function  value  for  all  260  problem  instances.  The  presented  greedy 
heuristics-based  algorithm  looks  very  promising  in  solving  some  real  life  preventive 
maintenance  scheduling  problems. 

Future  work  includes  the  addition  of  another  objective  to  the  algorithm:  the  effect  on 
system  reliability  at  iteration  j.  Currently,  only  a  cost  parameter  is  considered  when 
determining  the  earliness  or  tardiness  of  a  particular  component  maintenance  operation 
when  coordinating  system  maintenance.  However,  it  is  hypothesized  that  a  system  reliability 
objective  may  change  the  maintenance  schedule,  particularly  when  the  system  schedule 
suggests  that  some  components  be  maintained  after  their  recommended  maintenance  times 
(tardiness),  potentially  resulting  in  an  undesired  system  reliability. 
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Abstract 

A  commonly  cited  criticism  of  the  DoD  is  inefficiency  in  its  acquisition  process  that  leads  to  a 
high  potential  for  waste.  The  purpose  of  this  study  is  to  explore  whether  the  DoD’s 
institutional  setting  and  related  bureaucratic  structure  prohibit  leaders  and  policymakers  from 
effectively  implementing  private  sector  best  practices  related  to  strategic  sourcing,  especially 
demand  management.  Demand  management  requires  an  organizational  mindset  supporting 
the  governance  of  production  and  consumption  within  a  commodity  group.  A  qualitative,  case 
study  research  methodology  was  used  to  explore  whether  the  DoD’s  institutional  framework 
permitted  the  utilization  of  strategic  sourcing  processes,  such  as  demand  management. 
Gortner,  Mahler,  and  Nicholson’s  theoretical  framework  and  related  argument  that  public  and 
private  sector  organizations  differ  from  each  other  according  to  three  distinct  mediums  (legal, 
economic,  and  political)  was  applied.  Interview  data  and  document  artifacts  were  fractured 
and  coded,  then  grouped  into  categories  using  a  modified  grounded  theory  strategy.  Key 
findings  suggest  that  the  DoD’s  current  acquisition  structure  permits  a  limited  application  of 
demand  management  and  the  private  sector’s  key  success  factors  given  certain  political, 
legal,  and  economic  modifications. 

Research  Questions 

Despite  the  many  urgings  and  initiatives  for  improved  acquisition  processes  and 
methods,  the  DoD  continually  fails  to  implement  acquisition  reform  measures  that  would 
produce  the  desired  change.  Specifically,  the  DoD  and  its  bureaucratic  aversion  to  change 
is  unable  to  adopt  commercial  best  practices.  Regarding  its  acquisition  of  commercial  goods 
and  services,  the  private  sector  best  practice  of  strategic  sourcing  remains  absent  from  the 
DoD’s  standardized  acquisition  practices  despite  the  fact  that  the  Office  of  Management  and 
Budget  (OMB)  mandated  its  use  in  May  2005.  As  such,  inefficient  and  tactical  acquisition 
processes  continue  to  produce  wasteful  spending  practices.  This  problem  negatively 
influences  the  American  taxpayer,  the  DoD,  and  the  customers  that  it  supports,  most  notably 
the  warfighter. 
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A  possible  cause  of  this  problem  is  the  outdated  structure  of  the  DoD’s  acquisition 
system  and  the  procurement  function’s  limited,  administrative  role  in  the  overall  acquisition 
process.  This  study  investigates  commercial  best  practices,  such  as  strategic  sourcing,  and 
the  government’s  limited  success  in  applying  them  could  assist  in  remedying  this  problem. 

The  purpose  of  this  study  is  to  explore  whether  the  DoD’s  institutional  setting  and 
related  bureaucratic  structure  has  prohibited  it  from  effectively  implementing  strategic 
sourcing  practices.  This  study  applies  the  theory  and  research  asserted  by  Chubb  and  Moe 
(1990)  to  determine  whether  the  findings — that  the  institution  itself  and  its  outdated 
bureaucratic  processes  are  the  root  causes  of  inadequate  performance — also  apply  to  the 
DoD  acquisition  system. 

In  order  to  pursue  why  the  DoD  is  unable  to  implement  strategic  sourcing  practices 
across  its  acquisition  platform,  we  propose  the  following  three  research  questions: 

1 .  To  what  extent  does  the  DoD  acquisition  structure  limit  its  ability  to  practice 
strategic  sourcing? 

2.  Given  certain  DoD  initiatives,  what  variables/modifications  were  instituted  that 
promoted  successful  strategic  sourcing  practices? 

3.  Is  it  possible  to  mirror  these  successful  examples  and  apply  them  on  an 
enterprise-wide  basis  across  the  DoD  acquisition  platform? 

Literature  Review 

To  appreciate  the  magnitude  of  the  study’s  research  questions,  it  is  first  necessary  to 
analyze  and  detail  seminal  literature  that  focuses  on  bureaucracies  and  organizational 
theories,  as  well  as  some  of  the  key  differences  among  the  public  and  private  sectors  as 
they  pertain  to  these  constructs.  These  two  overarching  themes  each  play  a  significant  role 
in  determining  whether  public  sector  agencies  and  departments  are  successful,  or  whether 
they  possess  the  potential  to  be  successful,  in  adopting  private  sector  practices. 

Following  this  analysis  and  prior  to  exploring  strategic  sourcing  articles  from 
academic  journals,  a  thorough  examination  of  successful  strategic  sourcing  practitioners 
offers  insight  into  lessons  learned,  critical  success  factors,  and  other  related  details.  This 
portion  of  the  literature  review  highlights  which  strategic  sourcing  practices,  traits,  and 
components  have  proven  to  work  and  which  may  or  may  not  be  transferable  from  the  private 
to  the  public  sector. 

Classical  Literature 

Theorist  Herbert  Simon  (1997)  eloquently  detailed  the  importance  of  organizations  in 
his  landmark  work  Administrative  Behavior: 

Organization  is  important,  first,  because  it  provides  the  environments  that 
mold  and  develop  personal  qualities  and  habits.  Organization  is  important, 
second,  because  it  provides  those  in  responsible  positions  with  the  means  for 
exercising  authority  and  influence  over  others.  Organization  is  important, 
third,  because,  by  structuring  communications,  it  determines  that 
environments  of  information  in  which  decisions  are  taken.  We  cannot 
understand  the  inputs  or  outputs  of  executives  without  understanding  the 
organizations  in  which  they  work.  Their  behavior  and  its  effects  on  others  are 
functions  of  their  organizational  situations,  (p.  18) 

Despite  the  increasing  literature  and  focus  on  how  bureaucracies  are  changing, 
traditional  bureaucracies  continue  to  run  the  federal  government  and,  as  such,  they  should 
be  analyzed  and  studied  to  measure  their  impact  on  government  operations.  Assuming  that 
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Simon’s  (1997)  theory  that  organizations  affect  the  inputs  and  outputs  of  those  who  work 
within  them,  ignoring  organizations  makes  any  study  focusing  on  government  processes 
and  initiatives  incomplete. 

Simon  (1997)  asserted  a  clear  distinction  between  administrators  and  the  economic 
man.  Administrators,  according  to  Simon  (1997),  satisfice  rather  than  maximize,  implying 
that  they  can  make  decisions  without  knowing  or  ascertaining  all  the  facts  (p.  119).  Simon 
supported  this  theory  with  decades  of  management  and  human  behavior  observation  and 
research.  If  one  were  to  accept  Simon’s  assertion  that  administrators  act  in  a  satisficing 
manner,  then  administrators  can  and  will  make  decisions  with  established  rules.  Simon 
(1997)  characterized  these  rules  as  “relatively  simple  rules  of  thumb  that  do  not  make 
impossible  demands  upon  their  capacity  for  thought”  (p.  1 1 9).  Perhaps  this  characterization 
best  explains  the  myriad  of  rules  that  Wilson  (1989)  detailed  at  length  that  continue  to  run 
the  American  bureaucracy. 

Many  of  the  bureaucratic  tenants  detailed  by  Weber  (1 922)  and  Wilson  (1989) 
continue  to  dictate  the  composition  and  character  of  modern  bureaucracies,  including  the 
DoD,  which  remains  the  largest  department  within  the  executive  branch  in  both  budget  and 
population.  Academics  and  practitioners  alike  have  challenged  the  usefulness  and  efficiency 
of  traditional  bureaucracies  over  the  past  century,  and  these  critiques  and  suggestions 
warrant  consideration  in  view  of  the  changing  demands  and  expectations  placed  upon  these 
organizational  entities. 

Although  the  claim  that  Weber  (1922)  is  the  founder  of  bureaucracy  lies  outside  the 
scope  of  this  research  project,  it  is  safe  to  label  him  as  the  first  known  academic  to  define 
bureaucracy’s  attributes  and  promote  their  use.  Andreski  (1984)  asserted  that  Weber  was 
the  first  to  recognize  the  inevitable  bureaucratization  of  modern  governments  and  nation 
states,  which  is  one  of  the  most  significant  predictions  in  the  field  of  public  administration. 
Weber  (1922)  outlined  the  basic  characteristics  of  a  bureaucracy,  focusing  on  strict  and 
ordered  rules,  hierarchy  among  employees,  written  documents  that  guide  the  management 
of  the  modern  office,  managers  who  were  recognized  experts,  and  office  management  who 
followed  general  rules,  which  can  be  learned  (pp.  50-51). 

Bureaucracy  and  rules  complement  one  another.  Typical  connotations  of 
bureaucracies  initially  generate  images  regarding  rules  and  regulations,  a  concept  that 
Weber  (1922),  and  nearly  all  theorists  who  touch  on  bureaucracy,  detailed  at  length.  Wilson 
(1989)  asserted,  “The  United  States  relies  on  rules  to  control  the  exercise  of  judgment  to  a 
greater  extent  than  any  other  industrialized  democracy”  (p.  342). 

The  DoD,  perhaps  more  than  any  other  executive  department  in  the  United  States 
federal  system  due  to  its  size  and  complexity,  contains  a  seemingly  infinite  set  of  rules  and 
regulations.  For  example,  the  rules  that  guide  the  DoD  procurement  processes  are  divided 
into  several  layers,  including  the  federal  acquisition  regulations;  the  DoD  FAR  supplement; 
the  DoD  procedures,  guidance,  and  information;  as  well  as  a  host  of  local  agency 
procedures,  policies,  and  regulations.  As  of  July  1, 2010,  the  FAR  and  DFARS  alone  are 
2,074  and  954  pages,  respectively.  These  exhaustive,  overlapping  rules  do  not  include  the 
procurement  and  acquisition  rules  put  forth  by  the  OMB  in  the  form  of  directives,  policy 
memos,  and  circulars,  for  example.  A  contracting  officer  (CO)  in  the  DoD  requires  several 
years  of  both  training  and  on-the-job  experience  to  grasp  these  rules  well  enough  to  be 
granted  a  warrant  that  permits  an  officer’s  actions  independent  of  a  supervisor.  That  said,  all 
CO  actions  over  a  certain  dollar  threshold,  which  vary  depending  on  the  action  as  well  as 
the  agency,  must  still  go  through  a  legal  review  to  ensure  the  proper  application  of  these 
inherently  complicated  regulations,  further  illustrating  the  multitude  of  rules  and  regulations 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  574- 


employed  in  running  the  DoD.  Although  industry  has  its  own  review  processes,  it  typically 
falls  far  short  of  the  lengthy,  bureaucratic  process  imbedded  in  DoD  agencies. 

Regarding  the  topic  of  rules  and  the  order  that  they  produce,  Weber  (1922)  focused 
on  three  primary  tenants:  First,  regular  activities  are  fixed  in  such  a  manner  to  be  labeled  as 
official  duties.  Second,  the  authority  to  give  commands  is  strictly  outlined  and  followed 
(Weber,  1922,  p.  50).  Third,  only  those  with  the  qualifications  and  authority  are  employed 
and  are  done  so  in  a  continuous  manner.  Whether  these  primary  characteristics  of  how 
rules  are  implemented  and  sustained  remain  valid  in  today’s  environment  warrants  serious 
debate;  however,  regardless  of  one’s  opinion  on  this  particular  topic,  the  fact  remains  that 
some  degree  of  these  characteristics  is  evident  in  today’s  DoD. 

Osborne  and  Gaebler  (1992)  focused  several  publications  in  the  early  1990s  on 
public-  and  private  sector  differences  in  an  effort  to  highlight  the  need  for  change  and,  more 
broadly,  for  the  public  sector  to  begin  adopting  private  sector  practices  and  processes. 
Osborne  and  Gaebler  (1992)  in  Reinventing  of  America  stated,  “We  embrace  our  rules  and 
red  tape  to  prevent  bad  things  from  happening,  of  course.  But  those  same  rules  prevent 
good  things  from  happening.  They  slow  government  to  a  snail’s  pace”  (p.  Ill ).  There  are 
certainly  official  duties  in  the  DoD,  and  the  authority  to  perform  and  authorize  certain  actions 
are  clearly  spelled  out  in  a  host  of  policies  and  regulations.  These  actions  and  authorizations 
are  limited  to  those  officials  who  are  granted  explicit  authority  to  act  on  behalf  of  the  DoD. 
Focusing  on  the  topic  of  defense  acquisition  as  an  example,  each  defense  agency  has  its 
own  set  of  rules  and  policy  memorandums  that  carefully  and  explicitly  outline  the  authority 
levels  of  certain  employees  and  their  respective  positions.  At  a  higher  level,  each  defense 
agency  and  each  of  the  defense  services  must  follow  the  highly  integrated  DoD  Instruction 
5000.02,  which  is  the  lengthy  guide  that  details  the  multitude  of  approvals,  documents,  and 
authority  levels  that  serve  to  uniformly  control  the  acquisition  of  major  defense  acquisition 
programs.  In  short,  Weber’s  (1922)  intense  focus  on  the  rigidity  of  rules  within  a  bureaucracy 
is  a  characteristic  that  continues  to  flourish  in  the  DoD. 

Wilson  (1989),  citing  Weber  as  well  as  his  own  experiences,  supplemented  these 
thoughts  on  rules  with  his  own  assessment  regarding  the  gains  and  losses  produced  by  the 
rigid  application  of  rules  that  Weber  promoted.  Wilson  (1989)  asserted  that  the  difficulty  lies 
in 


striking  a  reasonable  balance  between  rules  and  discretion  is  an  age-old 
problem  for  which  there  is  no  ‘objective’  solution  ....  At  best  we  can  sensitize 
ourselves  to  the  gains  and  losses  associated  with  the  governance  by  rule 
rather  than  by  discretion,  (p.  342) 

In  the  world  of  defense  acquisition,  the  line  between  rules  and  discretion  is  anything 
but  concrete.  For  example,  nearly  all  the  rules  have  exemptions  specifying  that  government 
officials  can  employ  their  own  discretion.  However,  the  constantly  changing  political  realities 
and  pressures  frequently  determine  the  practitioner’s  ability  to  use  such  exemptions  and 
related  discretion. 

Demand-Side  Management 

Demand-side  management  involves  the  use  of  financial  incentives,  market 
mechanisms,  education,  efficiency  measures,  or  other  programs  to  modify  the  demand  for  a 
product  or  service  (Strengers,  2011).  The  demand  management  process  attempts  to  identify 
specific  sources  of  demand  so  that  procurement  organizations  can  ameliorate  the  risks 
associated  with  sources  of  demand.  Demand  management  attempts  to  control  when  or 
where  demand  occurs  in  order  to  match  it  efficiently  with  available  capacity  and  smooth 
highs  and  lows  of  demand  into  a  more  consistent  requirement  level  (Jack  &  Powers,  2009). 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  575- 


In  the  DoD,  demand  often  expands  to  the  level  of  funding  allotted  for  a  supply  or  service.  So 
rather  than  demand  driving  funding  and  procurement,  funding  drives  demand. 

Jack  and  Powers  (2009)  identified  examples  that  might  occur  in  the  health  care 
industry,  such  as  an  aging  and  growing  population,  the  increase  in  some  diseases  while 
others  are  reduced,  demand  for  new  treatments  and  therapies,  insurance  allowances  for 
procedures,  and  the  prevalence  of  managed  care. 

As  we  identified  previously,  organizations  are  impacted  by  politics  and  individual 
limitations  are  commonplace.  In  these  circumstances,  individuals  have  differences  in 
preferences  as  a  result  of  their  value  systems  and  bounded  rationality.  These  different 
preferences  impact  the  sourcing  of  products  and  services  (Cox,  Chicksand,  &  Ireland, 

2005).  These  differences  also  serve  as  major  sources  of  resistance  to  the  adoption  of 
enterprise-wide  sourcing  strategies. 

Public  and  Private  Sector  Organizational  Structures:  A  Comparative  Analysis 

Although  organizational  theory,  similar  to  studies  on  management,  generally  does 
not  differentiate  between  the  public  and  private  sectors,  there  are  common  characteristics, 
traits,  and  features.  The  subtle  differences  that  do  exist  can  and  do  have  profound 
consequences  when  attempting  to  implement  similar  processes  and  practices.  Academics 
Gortner,  Mahler,  and  Nicholson  (1989)  clarified  this  point  and  published  a  comprehensive 
text  that  focused  on  the  uniqueness  of  organizational  theory  as  it  applies  to  the  public 
sector.  Although  these  authors  admit  that  the  lines  that  once  separated  the  sectors  have 
somewhat  blurred  due  to  increasing  public  sector  laws  that  uniformly  apply  to  both  sectors, 
as  well  as  outsourcing,  the  public  sector’s  push  toward  commercial  practices,  and  so  on, 
they  still  convincingly  argued  that  the  public  sector  demands  its  own  focus  on  organizational 
theory. 

Gortner  et  al.  (1989)  asserted  three  fundamental  components  that  separate  public 
agencies  from  their  private  counterparts:  legal,  economic,  and  political  nature  and  roles. 
These  authors  argued  that  public  and  private  organizations  differ  “in  this  most  profound  way: 
It  is  the  business  of  public  bureaus  to  administer  the  law.  ...  Compliance  with  private  rules 
and  regulations  is  voluntary:  Non-compliance  in  the  public  sphere  may  result  in  coercion  or 
force”  (pp.  19-23). 

In  their  landmark  essay  “Comparing  Public  and  Private  Organizations,”  Rainey, 
Backoff,  and  Levine  (1976)  mirrored  the  thoughts  above  regarding  the  unique  legal 
differences  between  the  private  and  public  sectors  and  the  impact  that  these  differences 
creates.  Regarding  the  constraints  of  the  legal  system  that  applies  to  the  public  sector, 
Rainey  et  al.  (1976)  claimed  that  these  constraints  limit  the  public  manager’s  choices  as  to 
both  entry  and  withdrawal  of  certain  undertakings  (p.  238).  In  short,  the  legal  environment 
that  guides  the  public  sector  frequently  undermines  its  ability  to  freely  choose  its 
undertakings  and  related  practices  and  processes,  a  fact  that  is  rarely  noted  or  appreciated 
by  the  public  it  serves. 

The  economic  differences  between  the  public  and  private  sectors  can  be  succinctly 
summarized  by  the  fact  that  a  private  entity  is  largely  motivated  by  profit  whereas  a  public 
agency  must  blend  efficiency  with  political  and  legal  concerns,  and  mandates,  some  of 
which  were  discussed  previously.  For  example,  many  of  the  political  embargoes  and  trade 
restrictions  placed  by  the  United  States  were  instituted  due  to  political  concerns,  not  to 
enhance  profitability. 

Although  private  entities  do  not  operate  in  a  vacuum,  they  certainly  avoid  the  type  of 
interference  and  political  pressure  noted  by  Gortner  et  al.  (1989).  Allison’s  (1979)  landmark 
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presentation  comparing  the  sectors  and  highlighting  their  fundamental  differences 
specifically  noted  this  point.  Allison  (1979)  asserted,  “Government  managers  tend  to  have 
relatively  short  time  horizons  dictated  by  political  necessities  and  the  political  calendar”  (p. 
29).  Allison’s  point  adds  complexity  to  the  position  of  a  public  administrator,  who  must 
balance  political  demands  within  narrow  timeframes,  a  dangerous  combination  that 
inevitably  creates  hurried  and  frequently  inefficient  processes.  An  examination  of  the 
practitioner  literature  will  illustrate  the  key  success  factors  (KSFs)  that  are  inherent  in  any 
strategic  sourcing  program. 

Practitioner  Literature:  Key  Success  Factors 

The  overlapping  themes  regarding  what  practitioners  claimed  were  necessary 
ingredients  for  successful  strategic  sourcing  implementation  and  what  academia  is 
discovering  through  research  are  plentiful.  We  identified  a  host  of  repeating  suggestions  and 
criteria  for  a  successful  strategic  sourcing  program  emerged  from  the  literature.  These 
criteria,  or  key  success  factors,  can  be  categorized  into  the  following  high-level  headings: 
the  overall  status  of  the  purchasing  function,  effective  leadership  within  the  organization,  the 
ability  of  strategic  sourcing  teams  to  cross  functional  areas,  and  working  jointly  with 
suppliers  in  an  integrated  fashion  in  contrast  to  establishing  an  arms-length  relationship. 

This  final  KSF  includes  developing  suppliers  in  addition  to  simply  working  together.  The 
following  analysis  synthesizes  the  existing  literature’s  contribution  to  these  KSFs. 

Status  of  the  Purchasing  Function 

As  mentioned  by  Baldwin  et  al.  (2000),  Moore  et  al.  (2002),  Laseter  (1998),  and 
others,  practitioners  have  publicized  the  need  for  purchasing  to  cease  its  stereotypical  role 
of  serving  as  an  administrative  or  clerical  function.  Driedonks,  Gevers,  and  van  Weele 
(2010)  stressed  this  point  in  their  study  regarding  how  to  manage  the  effectiveness  of 
strategic  sourcing  teams  when  they  asserted,  “Although  things  have  changed  dramatically 
over  the  last  decades,  the  purchasing  profession  has  a  history  as  a  clerical  function”  (p. 

109).  Driedonks  et  al.  (2010)  claimed  that  the  ability  of  strategic  sourcing  to  create  a 
competitive  advantage  is  what  has  largely  raised  the  prominence  of  the  purchasing  function 
(p.  109). 

Because  the  DoD  has  not  altered  the  status  of  its  purchasing  function  since  it 
attempted  to  implement  strategic  sourcing  in  May  2005,  perhaps  because  it  does  not  have 
to  compete  in  the  marketplace  and  establish  any  type  of  competitive  advantage,  it  would  not 
satisfy  this  KSF.  Johnson  (2005),  whose  memorandum  implemented  government-wide 
strategic  sourcing  in  the  federal  sector,  did  not  establish  any  type  of  strategic  sourcing 
organizational  structure  or  make  any  mention  of  acquisition  process  and  procedures.  In 
short,  Johnson’s  OMB  memorandum  mandated  a  commercial  best  practice  but  maintained 
the  status  quo  in  terms  of  the  existing  status  and  role  of  the  purchasing  function. 

Kocabasoglu  and  Suresh  (2006)  enhanced  the  notion  of  increasing  the  status  of 
purchasing  when  they  asserted  that  successful  strategic  sourcing  implementation  depends 
on  purchasing  and  supply  managers  partaking  in  the  organization’s  strategic  processes  (p. 
7).  The  typical  DoD  framework — whereby  requirements  are  generated  and  provided  to  the 
purchasing  function  to  simply  administer  an  order — falls  far  shy  of  the  type  of  strategic, 
organizational  integration  that  Kocabasoglu  and  Suresh  label  as  a  KSF.  Schneider  (2011),  a 
professor  of  contract  management  at  the  Defense  Acquisition  University  (DAU),  asserted  the 
following  thoughts  regarding  the  status  of  the  purchasing  function  within  the  DoD: 

We  may  teach  the  acquisition  lifecycle  and  use  of  Integrated  Product  Teams 
but  the  reality  is  that  in  many  DOD  organizations,  procurement  is  not 
engaged  until  far  too  late  in  the  acquisition  planning  process.  This  means  the 
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value  add  of  the  business  advisor  is  minimized  and  it  is  no  surprise  that 
contracting  is  seen  as  more  of  an  administrative  paper-pushing  function. 
(Personal  communication,  March  28,  2011) 

Ogden,  Rossetti,  and  Hendrick  (2007)  confirmed  the  importance  of  this  KSF,  offering 
that  the  purchasing  literature  has  identified  “status  within  the  organization”  as  one  of  the  key 
determinants  of  purchasing’s  strategic  influence  (p.  4).  The  DoD’s  failure  to  break  through 
the  bureaucratic,  stove-piped  nature  of  its  acquisition  system  and  purchasing’s  continued 
administrative  role  will  inevitably  add  to  the  challenge  of  implementing  strategic  sourcing 
initiatives  and  practices. 

Effective  Leadership 

Wisma,  Schmidt,  and  Naimi  (2006)  asserted  that  significant  resources  need  to  be 
focused  on  leadership  in  order  to  properly  manage  the  inevitable  change  that  accompanies 
a  strategic  sourcing  initiative  (p.  174).  Wisma  et  al.  (2006)  argued  that  such  an  initiative 
without  an  effective  leader  to  manage  the  significant  change  that  stems  from  executing 
strategic  sourcing  practices  will  surely  lead  to  failure.  Although  Johnson’s  (2005)  OMB 
memorandum  directed  agencies  to  implement  strategic  sourcing  was  addressed  to  senior 
leaders  within  the  executive  departments — the  chief  acquisition  officers,  chief  information 
officers,  and  chief  financial  officers — there  was  no  guidance  on  how  to  lead  the  inevitable 
change  that  this  initiative  would  create.  Further,  the  leaders  who  received  this  OMB  tasking 
were  to  do  so  in  a  minor,  part-time  capacity,  further  emphasizing  the  weakened  approach 
employed  by  the  OMB  in  instituting  this  commercial  best  practice. 

By  comparison,  the  private  sector  typically  hires  experts  with  leadership  skills  whose 
sole  focus  is  to  drive  successful  strategic  sourcing  initiatives.  In  reviewing  the  organizational 
structures  and  leadership  roles  of  private  sector  firms  that  have  experienced  successful 
strategic  sourcing  programs,  it  turns  out  that  their  leaders  are  focused  on  the  primary 
mission  of  ensuring  that  their  programs  exceed  the  established  goals  and  metrics.  Klein 
(2004)  included  this  approach  as  one  of  the  three  key  steps  to  excellence.  Klein  (2004)  cited 
Prudential  as  a  best-in-class  case  study  in  his  research  and  stated  how  its  strategic 
contracts  manager  led  the  needed  change  (p.  24).  This  senior-level  executive  focused  on 
driving  effective  strategic  sourcing  practices  in  the  company.  This  type  of  focus  is  not  only 
lacking  but  is  frequently  altogether  absent  in  the  DoD  environment. 

The  federal  government’s  approach  was  to  add  this  challenging  initiative  to  the 
countless  other  duties  assigned  to  their  senior  leaders,  and  to  do  so  with  a  flat-lined  budget. 
In  this  instance,  the  federal  government  reverted  to  the  hierarchical  structure  and  tasked  the 
senior-most  leaders  who  delegated  down  to  their  subordinates  in  hopes  of  some  level  of 
progress  (Johnson,  2005).  Referencing  Simon’s  (1997)  thoughts,  the  outputs  of  executives 
cannot  be  understood  without  understanding  the  organizations  in  which  they  work  (p.  18). 
Regarding  the  implementation  of  strategic  sourcing  in  the  federal  sector,  the  important  trait 
of  leadership  was  handled  in  the  bureaucratic  fashion  by  tasking  and  delegating  in  lieu  of 
the  commercial  approach  of  hiring  dedicated  leadership  to  ensure  that  the  proper  level  of 
focus  and  energy  supported  the  initiative  (Johnson,  2005).  It  is  not  a  failure  of  leadership  but 
an  organizational  failure  that  best  explains  the  outputs  of  the  federal  and,  more  specifically, 
the  DoD  leaders.  This  possibility  should  be  kept  in  mind  moving  forward. 

Leadership’s  impact  on  strategic  sourcing  has  been  studied,  albeit  only  with  private 
sector  data.  Hult,  Ferrell,  and  Schul  (1998)  examined  the  impact  of  leadership  on  a  set  of 
individual  purchase  outcomes  related  to  the  sourcing  process.  Hult  et  al.  (1998)  studied 
leadership’s  impact  on  an  organization’s  purchasing  cycle  times  and  relationship 
commitments,  both  critical  measurement’s  in  assessing  an  organization’s  success  in  a 
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strategic  sourcing  environment.  Although  Hult  et  al.  (1998)  divided  leadership  into  multiple 
categories,  the  underlying  hypothesis  that  leadership  impacts  purchasing  outcomes  was 
confirmed  in  their  study. 

This  is  not  to  imply  that  effective  leadership  alone  ensures  that  strategic  sourcing 
initiatives  experience  success.  For  example,  when  Hawkins,  Randall,  and  Wittmann  (2009) 
researched  factors  contributing  to  the  use  of  reverse  auctions,  a  tool  frequently  used  by 
strategic  sourcing  teams,  they  illustrated  that  leadership  was  not  a  contributing  factor. 
Hawkins  et  al.  (2009)  revealed  that  leadership  in  their  study  only  proved  to  be  marginally 
significant  and  negatively  related  to  the  use  of  reverse  auctions  (p.  65).  Although  still 
marginally  effective  and  considered  by  most  to  be  a  critical  part  of  successful  strategic 
sourcing  practices,  leadership  alone  does  not  guarantee  positive  outcomes  from  a  strategic 
sourcing  perspective. 

Cross-Functional  Representation 

The  internal  coordination  of  purchasing  with  other  functions  is  a  KSF  that  relies 
heavily  on  internal  communications.  Freytag  and  Mikkelsen  (2007)  stressed  the  importance 
of  this  KSF,  stating  that  the  managerial  challenge  of  managing  relationships  is  critical  in 
implementing  strategic  sourcing.  Freytag  and  Mikkelsen  (2007)  asserted,  “The  key  to 
success  is  that  all  parts  of  an  organization  cooperate,  and  that  no  part  of  the  organization 
passively  or  actively  shows  a  reluctant  attitude  to  handling  the  tasks”  (p.  189).  Internal 
communication  and  managing  relationships  across  the  DoD’s  vast,  stove-piped  acquisition 
system  is  difficult  to  effectively  execute. 

Acquisition  is  comprised  of  a  handful  of  job  series,  although  this  handful  varies  from 
agency  to  agency.  These  job  series,  ranging  from  contracting  officers  to  program  managers 
to  engineers  to  logisticians,  contain  their  own  training,  competencies,  and  skill  sets  that 
rarely  overlap,  thereby  exacerbating  the  limited  view  and  scope  of  a  DoD  acquisition  official 
(Defense  Acquisition  University  [DAU],  2010).  For  example,  DAU  is  responsible  for  teaching 
certification  courses  to  the  DoD  acquisition  workforce.  The  DAU  website  (2011a)  put  forward 
the  following  clarification:  “The  Defense  Acquisition  Workforce  Improvement  Act  required  the 
Department  of  Defense  to  establish  a  process  through  which  persons  in  the  acquisition 
workforce  would  be  recognized  as  having  achieved  professional  status”  (para.  1). 

Each  DAU  campus  is  divided  up  into  six  departments:  contract  management, 
logistics,  program  management,  systems  engineering,  business,  cost,  and  finance  (DAU, 
2010).  The  certification  courses  provided  by  the  DAU  rarely  overlap,  and  the  instructors 
employed  to  teach  them  typically  teach  only  within  their  limited  area  of  expertise.  This 
approach  serves  only  to  heighten  the  stove-piped  nature  of  the  DoD  acquisition  system, 
making  the  internal  coordination  of  purchasing  with  other  functions  altogether  impossible. 

Mookherjee  (2008)  studied  the  criticality  of  moving  toward  a  flatter  organization  as 
opposed  to  traditional  vertical  organizations  that  typically  define  government  bureaucracies 
if  an  organization  is  to  effectively  practice  strategic  sourcing.  Mookherjee  asserted  that 
companies  are  therefore  moving,  and  sometimes  being  forced,  away  from  the  classical, 
vertical  structures  toward  those  that  are  more  flexible  (p.  72).  The  current  DoD 
organizational  structure  and  its  stove-piped  nature  violate  the  tenants  of  flexibility  and 
horizontal  platforms  that  Mookherjee’s  research  endorsed. 

In  a  study  similar  to  Mookherje’s  (2008)  study,  Gopal,  Viniak,  and  Caltagirone  (2004) 
outlined  a  model  to  achieve  a  strategic  sourcing  that  relies  heavily  on  the  effective  use  of 
cross-functional  teams.  Gopal  et  al.  (2004)  asserted,  “The  project’s  success  depends 
heavily  on  the  team’s  formation.  Purchasing,  logistics,  operations,  engineering,  and  finance 
all  need  to  be  represented  on  the  team”  (p.  56).  Again,  considering  that  the  DoD  currently 
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hires,  trains,  and  works  according  to  functional  area,  each  particular  function  is  frequently 
ignorant  regarding  the  roles  and  responsibilities  of  their  counterparts  in  the  other  functional 
areas. 


Mills  (2010),  a  former  program  manager  in  defense-related  acquisition  and  current 
DAU  professor  of  program  management,  recently  asserted  the  following  list  of  barriers  that 
limit  the  DoD’s  ability  to  form  integrated  product  teams:  lack  of  empowerment,  unclear  goals, 
poor  leadership,  unreasonable  schedule,  insufficient  resources,  and  lack  of  commitment  (p. 
31 ).  This  translates  the  tasking  of  a  joint  cross-functional  team  into  a  monumental  challenge, 
at  least  for  the  DoD  bureaucracy. 

This  is  not  to  imply  that  the  DoD  has  not  long  been  warned  regarding  the  need  to 
shift  from  a  functional,  narrow  focus  toward  the  industry  standard  of  cross-functional  teams. 
For  example,  Dupray  (2005),  a  contracting  officer  for  the  U.S.  Navy,  detailed  the  need  to 
transform  the  DoD’s  stove-piped  approach  to  acquisition  and  move  from  a  functional 
approach  toward  a  joint,  strategic  approach  (p.  8).  Despite  the  OMB  policy  letters,  countless 
executive  reports,  and  supporting  literature  from  the  private  sector,  the  massive  bureaucracy 
that  is  the  DoD  requires  fundamental,  organizational  change  to  truly  shift  from  its  existing, 
single  functional  area  focus,  which  unfortunately  prohibits,  or  at  least  limits,  strategic 
sourcing  processes. 

Buyer-Supplier  Relationships 

Information  sharing  and  the  development  of  key  suppliers  are  two  of  Kocabasoglu 
and  Suresh’s  (2006)  KSFs  that  come  close  to  violating  the  ethical  standards  and  statutory 
regulations  that  guide  the  federal  sector’s  acquisition  system.  The  DoD  acquisition  system, 
from  a  purchasing  perspective,  is  guided  by  the  federal  acquisition  regulations,  which  place 
the  concepts  of  fairness  and  competition  above  these  KSFs,  regardless  of  their  importance 
in  executing  strategic  acquisition  practices.  For  example,  FAR  3.101-1  offered  the  following 
guidance: 

Government  business  shall  be  conducted  in  a  manner  above  reproach  and, 
except  as  authorized  by  statute  or  regulation,  with  complete  impartiality  and 
with  preferential  treatment  for  none.  Transactions  relating  to  the  expenditure 
of  public  funds  require  the  highest  degree  of  public  trust  and  an  impeccable 
standard  of  conduct.  The  general  rule  is  to  avoid  strictly  any  conflict  of 
interest  or  even  the  appearance  of  a  conflict  of  interest  in  Government- 
contractor  relationships.  (General  Services  Administration  [GSA],  2011, 
section  3) 

It  is  easy  to  decipher  why  contracting  officers  are  hesitant  to  establish  long-term 
relationships  with  suppliers  because  the  FAR  clearly  articulates  that  complete  impartiality 
shall  guide  the  DoD  acquisition  process. 

The  current  bureaucratic  structure  of  the  DoD  and  the  lengthy  list  of  regulations  that 
guide  it  prohibit  certain  buyer-supplier  relationships  that  serve  as  common  practices  in  the 
private  sector  (GSA,  2011).  For  example,  FAR  15  details  how  contracting  officers  are  to 
treat  all  competing  industry  partners  fairly  to  ensure  equity  and  justice  in  the  federal 
contracting  process.  FAR  15.306(e)  specifically  stated,  “Limits  on  exchanges.  Government 
personnel  involved  in  the  acquisition  shall  not  engage  in  conduct  that  favors  one  offeror  over 
another”  (GSA,  2011).  When  these  rules  are  violated,  the  losing  contractor  in  a  competitive 
process  has  the  legal  right  to  protest  the  government’s  decision. 

FAR  33,  which  covers  protests,  disputes,  and  appeals,  is  dedicated  to  outlining  the 
processes  and  procedures  afforded  to  contractors  for  submitting  protests  when  it  suspects 
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that  the  government  violated  the  regulations.  For  example,  Northeast  Military  Sales  (NEMS) 
protested  the  Defense  Commissary  Agency’s  decision  to  award  to  another,  competing  firm. 
NEMS  won  the  protest  and  the  Defense  Commissary  Agency  had  to  restart  its  entire 
acquisition  process  because  it  failed  to  properly  follow  the  rules  regarding  the  engagement 
of  industry  suppliers.  In  its  ruling,  the  GAO  (201 1 )  asserted,  “NEMS  broadly  challenges  the 
agency’s  technical,  past  performance,  and  price  evaluations,  as  well  as  the  adequacy  of 
discussions.  We  sustain  the  protest”  (“Decision”  section,  para.  1).  The  reference  to 
discussions  in  the  GAO  decision  highlights  the  government’s  continued  failure  to  properly 
engage  with  industry.  This  recent  case  illustrates  the  complexity  involved  regarding  the 
DoD’s  use  of  this  particular  KSF. 

In  line  with  KSFs  that  promote  successful  strategic  sourcing  implementation,  Towers 
and  Song  (2010)  provided  that  long-term  purchasing  arrangements  are  necessary  between 
buyer  and  supplier  to  ensure  a  strategic  relationship  that  will  lead  to  effective  strategic 
sourcing  practices  (p.  542).  There  are  a  host  of  bureaucratic  hurdles  that  make  this  KSF 
difficult  to  achieve,  including  the  budgetary  system  that  funds  DoD  acquisitions,  the 
restrictions  regarding  the  use  of  multiyear  funds,  the  current  administration’s  intense  focus 
on  increasing  competition,  and  so  on.  For  example,  the  DoD  uses  various  categories  of 
money  to  fund  its  acquisitions,  all  of  which  have  strict  time  limits  regarding  when  they  can  be 
obligated  and  the  purpose  that  they  are  used.  Operations  and  maintenance  money,  for 
example,  must  be  used  within  the  year  that  it  is  provided  or  it  expires.  Further,  it  can  only 
fund  contracts  for  a  period  of  12  months,  further  emphasizing  the  point  regarding  the 
difficulty  for  the  DoD  to  establish  long-term  relationships  with  suppliers. 

To  illustrate  the  complexity  regarding  the  DoD’s  ability  to  establish  long-term 
relationships  with  suppliers,  consider  the  following  DoD  guidelines  established  in  the  DAU’s 
(2011b)  CON  216  Legal  Considerations  in  Contracting  course:  “Annual  appropriations  are 
made  for  a  specified  fiscal  year  and  are  available  for  new  obligation  only  during  the  fiscal 
year  for  which  made.  Routine  activities  of  the  federal  government  are,  for  the  most  part, 
financed  by  annual  appropriations”  (p.  206).  It  is  obvious  from  the  literature  that  the  long¬ 
term  relationships  promoted  by  strategic  sourcing  experts  far  exceed  the  one-year  time  limit 
that  is  typically  established  in  the  Appropriation  and  Authorization  Acts. 

In  the  summary  of  their  study,  Chan  and  Chin  (2007)  claimed  that  managing  and 
collaborating  with  suppliers  early  in  the  process  offers  companies  a  competitive  advantage 
(p.  1407).  This  competitive  advantage  escapes  the  largest  buying  entity  in  the  world 
because  of  its  bureaucratic  rules  that  continue  to  prohibit  it  from  applying  commercial  best 
practices. 

Although  strategic  sourcing  has  yielded  enormous  savings  for  industry  over  the  past 
20  years,  the  majority  of  strategies  focus  on  the  interactions  with  suppliers  through  contract 
award.  The  management  of  the  strategy  in  the  post-award  phase  has  had  comparatively 
little  attention  paid  to  it.  This  is  a  reflection  of  contracting’s  perceive  administrative  role 
(Hughes  &  Wadd,  2012). 

KSFs  Summary 

The  four  KSFs  detailed  in  the  previous  sections  are  all  limited  when  considering  the 
existing  institutional  structure  that  guides  the  DoD  acquisition  system.  Although  both 
scholars  and  practitioners  may  argue  over  which  KSF  is  most  critical  to  an  organization  that 
is  implementing  or  practicing  strategic  sourcing,  the  overlapping  themes  were  constant. 
Thawiwinyu  and  Laptaned  (2009)  executed  a  detailed  study  on  the  impacts  of  strategic 
sourcing  on  supply  chain  performance  management.  In  their  literature  review,  they  asserted 
the  following  as  the  main  elements  of  strategic  sourcing:  strategic  elevation  of  the 
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purchasing  function,  internal  coordination  between  supplier  and  purchasing,  long-term 
relationships  with  suppliers,  and  supplier  involvement  in  planning  and  design.  Their 
assessment  assists  in  validating  the  fact  that  these  elements  or  KSFs  are  constant  and 
need  to  be  massaged  into  the  DoD  institutional  setting  if  the  DoD  expects  to  realize  the 
utilization  of  strategic  sourcing  processes. 

Thawiwinyu  and  Laptaned  (2009)  asserted,  “Firms  that  implement  strategic  sourcing 
experience  significant  improvement  in  their  supply  chain  performance  management, 
specifically  in  terms  of  responsiveness  and  satisfaction  of  customer”  (p.  20).  In  an  era  of 
budget  cuts,  multiple  war  efforts,  and  overall  economic  uncertainty,  the  DoD  should  focus  its 
efforts  mightily  on  how  to  properly  implement  strategic  acquisition  practices  so  its  customers 
(the  warfighter  as  well  as  the  taxpayer)  can  experience  increased  customer  satisfaction, 
however  that  might  be  defined  (e.g.,  increased  savings,  better  service,  decreased  delivery 
times,  etc.).  Further,  the  potential  regarding  positive  social  change  associated  with  the 
reallocation  of  financial  resources  is  tremendous  serving  to  heighten  the  demand  for 
strategic  sourcing  in  the  DoD  and,  more  broadly,  in  the  public  sector. 

Research  Methods 

Considering  the  research  questions  and  their  exploratory  nature,  combined  with  the 
fact  that  the  academic  literature  revealed  little  information  on  strategic  sourcing  in  the  public 
sector,  a  case  study  methodology  is  the  best  approach  for  this  study.  Recognizing  that  the 
literature  on  strategic  sourcing  in  the  public  sector  is  limited  illustrates  that  this  truly  is  an 
exploratory  study.  This  research  effort  focuses  on  the  United  States  Air  Force  and  its 
Strategic  Sourcing  Program  Management  Office  (SSPMO),  referred  to  as  the  Air  Force’s 
Enterprise  Sourcing  Group  (ESG).  To  properly  assess  the  impact  of  how  the  DoD  could 
adjust  its  existing  organizational  structure  to  better  promote  strategic  sourcing  practices,  a 
program  that  has  illustrated  progress  in  this  realm  naturally  had  to  serve  as  the  case  study. 
Additionally,  if  the  research  question  of  whether  a  modified  strategic  sourcing  program  could 
be  applied  on  an  enterprise  level  within  the  DoD  is  to  be  explored,  a  program  that  has 
illustrated  the  ability  to  execute  strategic  sourcing  within  the  existing  regulations  had  to  be 
selected.  The  Air  Force’s  SSPMO  readily  meets  these  requirements. 

This  study  utilizes  both  an  archival  and  documentation  review,  as  well  as  interviews, 
in  an  effort  to  compile  as  much  data  as  possible  and  to  ensure  a  comprehensive, 
triangulated  approach.  The  archival  records  and  documentation  review  were  analyzed  by 
the  Gortner  et  al.  (1989)  framework  that  characterizes  public  and  private  sector  differences 
along  three  distinct  mediums:  legal,  economic,  and  political  (see  Table  1).  The  interviews 
were  conducted  along  the  private  sector’s  KSFs  that  emerged  in  the  literature  review. 
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Table  1.  Air  Force  Strategic  Sourcing  Archival  Records  and  Documentation 
Mapped  to  the  Gortner  et  al.  (1989)  Framework  Categories 


Air  Force  archival  record  or  documentation 

Gortner  et  al.  (1989)  framework 
(Legal/Economic/Political) 

IG5 3 07. 104-93  AF  Strategic  Sourcing  and 

Commodity  Council  Guide  (U.S.  Air  Force,  2010a) 

Legal 

Charter  for  the  Air  Force  Civil  Engineering 

Commodity  Council  (U.S.  Air  Force,  2010c) 

Legal/Political 

Charter  for  the  Air  Force:  Air  Force  Medical  Service 
Commodity  Council  (U.S.  Air  Force,  2010b) 

Legal/Political 

Charter  for  the  Air  Force  Furnishings  Commodity 
Council  (U.S.  Air  Force,  2010d) 

Legal/Political 

Strategic  Sourcing  Task  Group  Final  Report  Briefing 
to  the  Defense  Business  Board  (DBB,  2011) 

Legal/Economic/Political 

ESG’s  (2011)  Strategic  Alignment  and  Deployment 
Briefing  to  the  Air  Force  Materiel  Command 

Legal/Economic/Political 

ESG  Strategic  Sourcing  Briefing  to  the  Air  Force 
Materiel  Command  (AFMC;  Knipper,  2011) 

Legal/Economic/Political 

State  of  the  ESG/Follow-up  to  AFMC/CA  Briefing  to 
the  Air  Force  Materiel  Command  (Shofner,  2011) 

Legal//Political 

ESG  Strategic  Planning  Charters 

Legal/Economic/Political 

The  strength  of  this  study  lies  in  the  fact  that  the  methodology  aligns  with  the 
research  questions  that  are  guiding  it.  The  approach  to  triangulation  assures  that  the 
research  questions  are  explored  and  supported  by  a  myriad  of  viewpoints  and  data  sources, 
thereby  strengthening  the  data  and  related  analyses. 

We  conducted  a  rigorous  case  study  using  a  single  unit  of  analysis,  the  United 
States  Air  Force  SSPMO,  with  multiple  cases,  which  are  the  three  commodity  councils.  In 
this  case  study  we  incorporated  three  data  sources,  and  synchronized  the  results  of  the 
documentation  and  archival  records  analyses,  executed  through  the  Gortner  et  al.  (1989) 
organizational  framework,  with  follow-on  interviews  of  the  employees  running  the  referenced 
commodity  councils.  This  comprehensive  approach  to  triangulation,  along  with  the  detailed 
steps  to  minimize  or  eliminate  the  validity  threats,  ensures  that  the  proposed  research 
questions  are  pursued  with  academic  rigor. 

Research  Findings  and  Impact  on  DoD  Acquisition 

The  archival  records  and  documentation  review  clearly  provides  invaluable  insight 
into  this  study’s  research  questions.  That  portion  of  the  study  highlighted  how  the  DoD’s 
bureaucratic  structure  does  not  limit  an  agency’s  ability  to  launch  a  strategic  sourcing 
program,  which  alone  validates  the  usefulness  and  contributory  nature  of  this  research 
effort.  The  extent  to  which  it  may  limit  an  agency’s  ability  to  successfully  employ  strategic 
sourcing  practices  remains  somewhat  less  defined. 

In  the  best  of  circumstances,  demand  managers  must  address  a  number  of 
challenges  that  occur  when  meeting  requirements  including  product  over  specification, 
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premature  or  tardy  establishment  of  specifications,  frequent  changes  in  specification,  poor 
or  non-existent  demand  information,  fragmentation  of  spend,  maverick  buying,  agency 
politics,  and  the  risk-averse  nature  and  culture  of  the  organization  (Cox,  Chicksand,  & 
Ireland,  2005).  Our  findings  indicate  that  each  of  these  challenges  is  exacerbated  by  the 
DoD  bureaucracy. 

Product  overspecification  occurs  when  individual  users  are  permitted  to  define 
requirements,  rather  than  a  team  of  experts  representing  the  enterprise  requirements. 
Premature  establishment  of  specifications  may  limit  the  sourcing  team’s  ability  to  negotiate  a 
solution  with  suppliers,  or  to  identify  current  state-of-the-art  or  alternative  best-value 
solutions.  Changes  in  specification  make  the  organization  susceptible  to  changes  in  price 
from  suppliers  that  may  increase  total  cost.  Poor  demand  information  deprives  the 
organization  of  making  informed  decisions  and  puts  them  at  a  negotiating  disadvantage. 
Spend  that  is  dispersed  to  more  vendors  than  necessary  increases  transaction  cost; 
however,  the  political  environment  that  DoD  buyers  work  in  requires  attention  with  regard  to 
small  business  vendors.  The  risk-averse  nature  of  DoD  buying  organizations  is  exacerbated 
by  the  penalties  assessed  when  an  innovative  strategy  proves  unsuccessful. 

The  ESG’s  ability  to  work  around  the  legal,  political,  and  economic  differences  to 
continue  to  push  its  program  forward  is  a  testament  that  the  limitations  imposed  by  the  DoD 
bureaucracy  can  be  overcome  or,  perhaps  more  appropriately,  circumvented.  The  ESG  was 
able  to  modify  certain  processes  and  regulations  to  work  around  barriers.  This  seemingly 
simple  achievement  highlights  the  greater  impact  of  applying  the  ESG’s  approaches, 
processes,  lessons  learned,  and  achievements  toward  other  DoD  agencies  and  services. 
Additionally,  the  idea  that  an  enterprise-wide  DoD  strategic  sourcing  program  could  take 
root  is  within  the  realm  of  the  possible,  a  thought  that  traditional  bureaucrats  would  likely 
dismiss. 

The  interview  data  illustrate  that  the  Air  Force  ESG,  operating  under  the  same  rigid 
DoD  acquisition  structure  as  its  peers,  was  able  to  explore  and  achieve  varying  degrees  of 
success  with  respect  to  the  KSFs  that  emerged  in  the  literature.  The  research  results 
detailed  in  this  study  revealed  that  the  ESG  was  able  to  significantly  advance  the  status  of 
the  purchasing  function  as  well  as  the  concept  of  cross-functional  representation  on  the 
strategic  sourcing  commodity  councils.  Although  progress  was  evident  in  the  data,  this  does 
not  imply  that  the  ESG  reached  a  maturity  level  equal  to  its  private  sector  peers.  Some  of  its 
shortcomings  (e.g.,  bureaucratic  processes  that  result  in  inefficient  reporting  structures  and 
limited  number  of  SMEs  on  the  commodity  councils)  are  likely  related  to  the  organizational 
structure  and  the  related  limitations  that  this  produces. 

The  KSFs  focusing  specifically  on  effective  leadership  and  engaging  suppliers  in  a 
collaborative  manner  have  witnessed  success  to  a  lesser  extent  than  the  aforementioned 
KSFs.  The  limited  but  nonetheless  noteworthy  ESG  successes  in  these  specific  KSFs  are 
easier  to  tie  to  the  DoD’s  bureaucratic  structure.  For  example,  leadership  within  the  DoD  is 
still  focused  on  tactical  and  narrow-minded  approaches  that  tend  to  focus  on  supporting  only 
those  initiatives  that  leaders  either  own  or  are  at  least  personally  invested  in  versus  the 
strategic  approach  of  supporting  initiatives  that  benefit  the  organization.  Similarly,  the 
bureaucratic  culture  and  rules  surrounding  fairness  and  competition  makes  the  full 
realization  of  the  final  KSF  that  focuses  on  supplier  collaboration  difficult,  if  not  impossible, 
to  witness  in  the  short  term. 

This  research  provides  a  solid  baseline  for  both  the  ESG  and  its  DoD  counterparts  to 
recognize  what  can  be  initially  achieved,  which  KSFs  are  more  difficult  and  may  require 
additional  support  or  changes,  and  which  approaches,  measures,  and  processes  have 
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proven  most  effective.  Although  more  detailed  answers  to  these  questions  will  only  be 
possible  overtime,  this  study  has  produced  significant  and  worthwhile  information  that  will 
prove  useful  to  the  Air  Force  and,  more  broadly,  the  DoD  as  it  continues  to  move  toward 
more  strategic  sourcing  and  related  acquisition  practices. 

Although  the  DoD  acquisition  structure  limits  its  own  ability  to  efficiently  establish  and 
promote  strategic  sourcing  processes,  it  does  not  block  it  entirely.  ESG  employees 
expressed  confidence  in  the  fact  that,  over  time,  its  program  will  mature.  It  has  already  made 
significant  progress  in  two  years  within  a  bureaucratic  structure  that  has  solidified  its  cultural 
norms  over  the  past  50  years.  This  encouraging  thought  demands  additional  research, 
especially  in  light  of  the  positive  social  change  that  will  naturally  stem  from  the  DoD’s  ability 
to  realize  effective,  robust,  strategic  sourcing  practices  and  programs. 

This  research  effort  has  produced  obvious  demands  for  action  from  DoD  acquisition 
leadership,  related  policy  leaders,  and  practitioners  alike  (e.g.,  contracting  officers  and 
program  executive  officers).  The  fact  that  the  ESG  was  able  to  effectively  initiate  a  fully 
functioning  strategic  sourcing  program  management  office  within  the  boundaries  of  the 
same  bureaucratic  environment  that  guides  and  governs  its  peers  mandates  that  other  DoD 
agencies  and  Services  take  the  initiative  to  implement  strategic  sourcing  practices  and  reap 
the  myriad  benefits  that  it  offers. 

As  a  leader  in  this  particular  space,  the  Air  Force  ESG  should  meet  with  its  DoD 
counterparts  to  review  both  its  successes  and  failures.  It  should  explore  how  its  peers  can 
and  should  follow  in  its  footsteps  and  actively  partner  with  them  in  establishing  similar  PMOs 
at  their  respective  services  and  agencies.  The  ESG  should  also  record  a  baseline  for  its 
current  position  and  benchmark  it  against  its  goals  and  private  sector  peers  to  determine 
how  much  further  it  can  advance  its  current  progress  and  what  it  will  require  to  do  so.  The 
ESG  should  map  out  these  next  steps  for  the  critical  stakeholders  and  leadership  who  can 
and  should  help  it  achieve  its  goals.  In  lieu  of  begging  for  additional  leadership  support  and 
better  cross-functional  representation,  the  ESG  should  be  more  proactive  in  gaining  the  type 
of  leadership  and  stakeholder  support  that  will  endorse  these  supportive  measures  and 
drive  it  from  the  top  down.  These  types  of  aggressive  pursuits  pose  potential  gains  that 
more  than  justify  their  endorsement. 

Significance  of  This  Study 

The  purpose  of  this  exploratory  case  study  research  effort  was  to  initiate  an 
investigation  into  whether  the  DoD’s  bureaucratic  structure  limits  its  ability  to  practice 
strategic  sourcing.  The  resulting  data  illustrate  that  although  the  structure  may  have  a 
limiting  effect,  it  certainly  does  not  prevent  implementation  and  potential  success  in  a 
number  of  strategic  sourcing  processes  and  areas.  The  legal,  environmental,  and  political 
differences  between  the  public  and  private  sectors  did  not  prevent  the  Air  Force’s  ESG  from 
successfully  launching  its  SSPMO  and  from  achieving  initial  success  stories.  This  research 
should  ultimately  prove  valuable  for  its  discoveries  that  strategic  sourcing  success  is 
possible  within  the  DoD,  that  the  ESG’s  success  is  likely  transferable  to  other  DoD  services 
and  agencies,  and  that  the  KSFs  required  for  success  can  be  implemented  and  achieved,  to 
some  degree,  within  the  DoD’s  bureaucratic  environment. 

We  are  pragmatic  enough  to  stipulate  that  the  bureaucratic  and  risk-adverse  nature 
of  DoD  procurement  will  change  little  in  the  near  future.  As  a  result,  the  routinized  tasks  that 
are  currently  repeated  with  each  transaction  that  procurement  organizations  conduct  will 
also  continue.  We  propose  that  strategic  sourcing  can  make  a  significant  contribution  in  this 
area  by  accomplishing  those  routinized  tasks  a  single  time  for  a  group  of  products  or 
services.  Such  a  strategy  will  yield  significant  process  cost  savings  for  the  enterprise. 
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Even  with  these  discoveries,  this  field  of  research  begs  for  additional  studies  and 
more  granular,  targeted  focus  areas.  For  example,  the  ESG  must  explore  how  to  better 
achieve  each  of  the  four  KSFs  within  DoD  regulations  if  it  seeks  to  mature  its  existing 
SSPMO.  Another  key  area  of  focus  for  the  ESG  should  be  to  better  understand  how  to 
restructure  its  acquisition  function  areas  in  the  hope  of  doing  a  better  job  at  achieving  the 
stated  KSFs.  The  ESG’s  peers  should  explore  and  study  how  they  can  mirror  the  ESG’s 
success,  improve  on  the  existing  lessons  learned,  and  team  with  the  ESG  to  drive  cross¬ 
agency  and  department  collaboration. 

Examining  the  broader  topic  of  private  sector  practices  within  the  federal  sector  is 
another  area  that  is  ripe  for  exploration.  This  strategic  sourcing  case  study  could  be 
expanded  to  examine  the  topic  of  commercial  best  practices  within  the  federal  sector  and 
what  is  (and  is  not)  possible  or,  perhaps,  what  is  (and  is  not)  warranted  or,  perhaps,  what  is 
(and  is  not)  necessary  in  these  challenging  fiscal  times. 

All  considered,  the  purpose  of  this  study  was  to  launch  the  necessary  investigation  of 
how  to  apply  efficient  commercial  practices  within  a  seemingly  rigid  bureaucratic 
environment.  This  research  is  meant  to  initiate  the  larger  and  much  more  focused,  follow-on 
research  that  is  hopefully  waiting  to  be  explored  by  other  scholar-practitioners  in  the  field  of 
public  administration. 
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Abstract 

The  benefits  of  strategic  sourcing  have  been  realized  by  private  industry  for  over  two 
decades.  Despite  the  compelling  business  case  presented,  the  adoption  of  strategic  sourcing 
tenets  in  government  procurement  has  been  slowed  by  a  lack  of  leadership  and  committed 
resources  (GAO,  2012).  We  believe  that  advancing  the  ability  to  identify,  capture,  and 
communicate  cost  savings  that  accrue  from  strategic  sourcing  activities  will  allow  government 
procurement  leaders  to  better  articulate  the  value  of  such  programs.  Enhanced 
communication  will  enable  leaders  to  pursue  the  appropriate  resources  to  sourcing  teams.  In 
order  to  tell  the  story  in  a  more  effective  manner,  leaders  must  understand  the  types  of  cost 
they  are  incurring  and  the  drivers  of  cost  that  they  can  impact,  and  they  must  ensure  that 
their  teams  take  credit  for  the  total  spectrum  of  cost  that  they  affect.  This  paper  examines  the 
various  types  of  savings  that  may  accrue  to  an  organization  pursuing  strategic  sourcing 
strategies  and  recommends  the  grouping  of  savings  into  rate,  process,  and  demand 
categories.  In  addition  to  introducing  the  types  of  cost,  examples  of  cost  and  scenarios 
whereby  organizations  have  achieved  cost  savings  are  presented. 

Introduction 

Strategic  sourcing  offers  a  myriad  of  practices,  models,  and  processes  that  are 
typically  targeted  at  a  specific  cost  driver  and/or  cost  pool.  Although  strategic  sourcing  and 
its  potential  impact  are  far  reaching,  it  consistently  aims  to  drive  efficiencies  and  savings 
across  an  organization.  After  briefly  detailing  some  of  the  potential  cost  savings  and 
efficiency  areas  across  the  supply  chain,  we  will  specifically  hone  in  on  an  organization’s 
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ability  to  affect  the  following  cost  groups:  rate,  process,  and  demand.  These  cost  groups  will 
be  defined,  explained,  and  supported  with  examples  to  illustrate  the  potential  impact  that 
sourcing  strategies  can  have  when  directly  applied  to  them.  In  the  current  fiscal  climate,  it  is 
imperative  that  any  DoD  procurement  strategy  be  supported  with  tangible  metrics  and  a 
calculated  return  on  investment  (ROI)  so  that  an  effective  business  case  can  be  made  to 
ensure  leadership  support  and  follow-on  execution.  The  recommended  approach  of  applying 
strategic  sourcing  strategies  targeted  to  achieving  efficiencies  in  the  rate,  process,  and 
demand  cost  groups  significantly  advance  procurement  leaders’  ability  to  develop  a 
compelling  and  comprehensive  business  case. 

Types  of  Strategic  Sourcing  Cost  Savings 

The  broad  spectrum  of  cost  in  the  supply  chain  includes  manufacturing, 
administration,  warehouse,  distribution,  capital,  and  installation  cost  (Pettersson  & 
Segerstedt,  2012).  By  considering  all  phases  of  the  supply  chain,  strategies  formulated  by 
strategic  sourcing  teams  influence  the  cost  drivers  in  each  of  these  cost  pools. 

Strategic  sourcing  activities  have  the  ability  to  impact  each  of  these  key  cost  areas. 
As  we  shall  discuss,  standardized  configurations  can  reduce  manufacturing  costs.  Bulk 
ordering  can  reduce  distribution  cost.  Just-in-time  delivery  can  reduce  or  eliminate 
warehouse  cost  and  capital  cost.  Enterprise-wide  contracts  can  reduce  installation  costs. 
Although  these  steps  may  help  reduce  external  costs,  focusing  on  the  procurement  activity 
itself  can  reduce  process  costs  in  the  administration  cost  pool. 

It  is  interesting  that  private  industry  organizations  have  dedicated  tremendous  effort 
and  focus  on  the  management  of  internal  cost  (Cokins,  2001).  However,  they  spend  much 
less  time  attempting  to  influence  the  internal  behavior  of  vendors  in  their  supply  chain  in  any 
way  other  than  price  negotiations.  To  the  small  extent  that  government  procurement  exerts 
influence  on  supplier  behavior,  it  is  constrained  to  proposal  and  contract  policy  and  small 
business  regulations.  In  the  converse  of  industry  behavior,  government  spends 
comparatively  little  effort  in  attempting  to  understand  and  enhance  their  internal  processes. 
Both  industry  and  government  can  benefit  from  improved  understanding  and  involvement  in 
supplier  behaviors  and  vendor  cost  drivers;  and  government  could  see  tremendous  value  in 
an  examination  of  the  internal  processes  and  cost  associated  with  procurement. 

Of  course  focusing  on  cost  “numbers”  is  of  limited  value.  As  Cokins  (2001)  put  it,  in 
describing  successful  cost  managers,  “You  do  not  really  manage  costs,  you  understand  the 
causes  of  cost”  (p.  28). 

The  recent  history  of  cost  management  has  utilized  several  different  tactics.  In  the 
1970s,  direct  product  profitability  (DPP)  was  utilized.  This  system  focused  on  the  costs 
associated  with  a  particular  product  (Cokins,  2001).  In  the  1980s,  total  cost  of  ownership 
(TCO)  emerged.  TCO  examined  the  entire  cost  to  acquire,  use,  and  dispose  of  an  item, 
rather  than  just  considering  the  purchase  price  (Ellram,  1994).  Activity-based  costing  (ABC) 
gained  popularity  during  this  time  as  well.  ABC  places  cost  in  categories  related  to 
organization  activities  or  objectives  (such  as  business  development  or  presentation 
preparation,  rather  than  aggregate  categories  such  as  labor  cost).  As  a  result,  use  of  ABC 
provides  insight  into  the  cost  of  specific  organizational  activities.  As  is  the  case  with  most  of 
these  cost  systems,  ABC  has  an  inward-looking  focus  (Cokins,  2001). 

From  a  strategic  sourcing  perspective,  leaders  should  be  focused  on  both  internal 
and  external  causes  of  cost.  Further,  the  causes  of  cost  of  concern  should  be  those  that  the 
strategic  sourcing  team  can  affect.  We  recommend  classifying  the  causes  of  addressable 
procurement  cost  into  three  distinct  cost  groups  that  can  be  impacted  by  sourcing  strategies: 
rate,  process,  and  demand. 
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Rate 


In  almost  every  case,  an  organization’s  first  efforts  to  implement  strategic  sourcing 
are  aimed  at  attempts  to  reduce  the  cost  paid  per  item  for  a  particular  good  or  service. 

These  initial  efforts  often  are  of  the  “leveraged  buying”  variety.  A  clear  example  of  leveraged 
buying  is  presented  in  the  scenario  wherein  consumers  purchase  50  rolls  of  paper  towels  in 
bulk  at  a  shopping  club  outlet  to  achieve  a  reduced  cost  per  item.  Leveraged  buying  allows 
an  organization  to  achieve  rate  cost  savings.  Simply  put,  the  cost  per  unit  paid  for  the  same 
product  or  service  is  reduced  by  developing  and  implementing  rate  savings  related 
strategies.  Implementing  leveraged  buying  strategies  can  achieve  quick  wins  for  the 
sourcing  organization  and  prove  particularly  successful  in  straightforward  commoditized 
product  or  service  groups.  However,  rate  savings  are  just  one  type  of  cost  that  organizations 
can  impact  through  strategic  sourcing. 

Process 

Although  the  savings  realized  through  rate  reductions  are  often  finite,  they  are  often 
substantially  realized  in  the  short  term.  The  more  complex  buckets,  process  cost  and 
demand  cost,  have  potentially  higher  savings  over  a  longer  period  of  time.  Process  cost  is 
the  cost  that  is  required  for  an  organization  to  buy  a  product  or  service.  This  cost  includes  all 
facets  of  the  procurement  process  from  requirement  definition  through  contract 
management  and  closeout.  In  organizations  utilizing  decentralized  buying,  similar  items  are 
purchased  in  small  quantities  at  many  locations  on  a  repeated  basis.  The  cost  for  each 
transaction  is  repeated  with  each  buy.  Organizations  can  reduce  this  cost  by  pursuing 
strategies  that  put  a  common  buying  process  in  place  and  allow  purchases  to  be  repeated  at 
multiple  locations  on  a  recurring  basis  (Reed,  Bowman,  &  Knipper,  2005). 

Consider  a  web-based  shopping  service  that  allows  customers  to  compare  prices 
and  load  buying  data  and  delivery  information  one  initial  time.  On  subsequent  visits,  the 
customer  might  simply  select  an  icon  to  purchase  the  same  item  again,  thus  reducing  the 
cost  in  time  and  personnel  required  to  complete  the  transaction.  In  federal  purchases, 
moving  away  from  single  transactions  to  utilizing  pre-negotiated  blanket  purchase 
agreements  or  multiple-award  contracts  can  reduce  the  front-end  labor  requirement  and 
streamline  the  buying  process.  In  fiscal  year  (FY)  2012,  the  DoD  conducted  14,263,469 
transactions  (accessed  at  usaspending.gov).  Potential  savings  from  process  cost  reduction 
by  eliminating  some  of  these  transactions  and  transitioning  from  complex  contract  execution 
to  ordering  off  strategic  vehicles  where  possible  can  yield  millions  of  dollars  in  process  cost 
savings. 

Demand 

A  third  type  of  potential  savings  is  demand  savings.  Demand  savings  focuses  on 
reducing  the  total  number  of  units  purchased.  Switching  from  incandescent  light  bulbs  to 
LED  bulbs  is  an  example  of  reducing  demand  cost.  By  seeking  out  solutions  that  reduce  the 
total  number  of  products  or  services  required  in  order  to  meet  the  mission,  the  DoD  can 
reduce  demand  cost.  Demand  cost  reduction  can  occur  in  multiple  cost  pools  depending  on 
the  item.  In  addition  to  the  item  procured,  it  could  also  include  maintenance  time,  inventory 
support,  and  other  logistics  cost  that  may  be  reduced  (Reed  et  al.,  2005). 

Examples  of  Cost  Savings  in  Air  Force  Strategic  Sourcing 

We  now  turn  to  an  example  of  an  Air  Force  (AF)  sourcing  strategy  to  illustrate  the 
placement  of  cost  causes  into  the  three  recommended  categories.  The  AF  Civil  Engineering 
Support  Agency  (AFSECA)  identified  the  conversion  of  taxiway  lights  to  LED  as  a  strategic 
sourcing  opportunity  in  2010.  The  AF  had  over  30,000  taxiway  lights,  which  illuminate  the 
edges  of  runways  and  taxiways  at  AF  bases.  One  third  of  AF  taxiway  lights  had  already 
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been  converted  to  LED  by  independent  bases.  However,  there  was  no  enterprise-wide 
approach  to  bulb  conversion.  As  a  result,  the  AF  was  not  leveraging  its  purchasing  power 
(rate-related  cost).  It  was  not  incurring  standardized  inspection,  electricity,  and  maintenance 
costs  (demand-related  causes  of  cost).  Finally,  it  was  inefficiently  conducting  procurement 
transactions  (process-related  causes  of  cost). 

In  early  2010,  the  newly  formed  Civil  Engineering  Commodity  Council  (CECC)  took 
on  the  challenge  to  strategically  source  LED  taxiway  lighting  with  an  enterprise-wide 
strategic  approach.  A  cross-functional  team  of  acquisition  and  operational  professionals  was 
formed.  Base-level  civil  engineers,  airfield  managers,  and  flight  operations  were  identified  as 
the  affected  requirement  owner’s  subject-matter  experts.  Program  managers,  data  analysts, 
and  contract  specialists  from  the  AF  Enterprise  Sourcing  Group  led  the  multi-functional 
team.  The  AF  completed  the  sourcing  strategy  in  October  2010  (Quinter,  2012).  We 
examine  the  forecast  reductions  in  causes  of  cost  in  the  next  section. 

Rate  Cost  Efficiencies 

The  rate  cost  efficiency  is  based  on  the  reduction  in  the  price  per  unit  paid  for  each 
LED  type  light.  With  more  than  10,000  lights  being  replaced  over  the  last  year,  energy 
savings  are  being  realized.  From  a  cost  savings  perspective,  since  being  awarded,  the 
contracts  have  been  utilized  to  provide  replacement  lights  at  18  AF  installations  in  10  states. 
Cost  savings  of  50-60%  were  anticipated,  based  on  past  spend  for  incandescent  lamps. 
Although  actual  cost  savings  are  still  being  calculated  for  the  last  quarter,  $300,000  in  cost 
savings  has  been  confirmed  (Quinter,  Wilkins,  Bell,  Bowling,  &  Tungate,  2012).  As  we  have 
discussed,  leveraging  buying  power  to  reduce  the  price  paid  per  unit  is  most  often  the  initial 
focus  for  sourcing  strategy  teams.  Although  these  savings  can  be  significant  at  the  outset, 
the  source  of  enduring  savings  is  in  understanding  and  affecting  the  causes  of  cost  in  the 
process  and  demand  categories. 

Process  Cost  Efficiencies 

By  having  a  centralized  contract  vehicle,  AF  buying  offices  can  now  place  orders  off 
existing  contract  vehicles  rather  than  creating  new  contracts  or  orders  for  each  purchase. 
This  reduces  the  amount  of  effort  required  to  acquire  taxiway  lighting.  The  AF  utilizes  a 
process  cost  model  to  calculate  process  cost  savings  based  on  the  number  and  type  of 
transactions  avoided  as  a  result  of  new  strategies. 

Demand  Cost  Efficiencies 

Demand  cost  efficiency  in  this  example  can  also  be  seen  as  anything  that  occurs  as 
a  result  of  the  strategy  that  reduces  or  changes  consumption  related  to  the  item.  In  this 
case,  two  primary  benefits  were  realized  by  the  AF  from  moving  to  LED  lighting.  The  first 
cause  of  cost  that  is  affected  by  the  strategy  is  energy  consumption:  The  LED  fixtures  are 
designed  to  use  60%  less  energy  than  conventional  lighting  (Quinter  et  al.,  2012).  The  AF 
validated  energy  savings  by  separately  metering  airfield  lighting  installed  at  one  AF  base 
and  using  the  achieved  reduction  calculation  as  the  per-unit  energy  savings  factor.  Total 
savings  are  calculated  using  an  aggregation  of  expected  energy  savings  multiplied  by  the 
number  of  units  installed. 

The  second,  cause  of  cost  affected  by  the  strategy  is  maintenance  labor  cost.  The 
LED  lights  have  an  average  life  expectancy  of  more  than  100,000  hours,  compared  to  the 
1 ,000  hours  provided  by  the  previous  incandescent  fixtures  (Quinter  et  al.,  201 2).  Due  to  the 
longer  life  of  the  LED  fixtures,  the  airfield  maintenance  (inspecting  and  replacing  burned  out 
bulbs)  is  dramatically  less.  Because  the  incandescent  bulbs  burn  out  and  need  to  be 
replaced  much  more  frequently  than  the  LED  fixtures,  which  are  virtually  maintenance  free, 
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the  task  of  maintaining  the  fixtures  is  all  but  eliminated  (except  due  to  damage).  Calculations 
utilizing  average  labor  rates  and  observed  labor  touch  times  were  used. 

Recommendations 

The  adoption  of  strategic  sourcing  by  government  has  been  slowed  by  many  factors. 
A  significant  barrier  is  that  procurement  organizations  continue  to  be  staffed  primarily  with 
single  function  workers  with  no  experience  in  strategic  procurement  principles  or  techniques. 
The  government  is  further  limited  by  a  focus  on  tactical  execution  of  one  requirement  at  a 
time  rather,  than  an  enterprise-wide,  strategic  perspective. 

We  acknowledge  that  organizational  inertia  in  government  is  too  powerful  a  force  to 
overcome  in  the  pursuit  of  changing  the  way  these  buying  organizations  behave.  Rather,  we 
suggest  the  establishment  of  new,  multi-functional,  multi-skilled  organizations  to  create  and 
execute  sourcing  strategies  for  the  enterprise.  Establishing  these  organizations  requires  a 
compelling  business  case  based  on  the  standardized  identification  of  potential  savings  that 
are  possible  through  strategic  sourcing.  Such  a  business  case  will  likely  demonstrate  a 
significant  ROI  relative  to  the  cost  required  to  establish  the  organization. 

We  recommend  developing  a  standardized  methodology  to  identify,  capture,  and 
communicate  cost  causes,  and  subsequent  savings  is  essential  to  securing  the  resources 
needed  to  implement  successful  sourcing  strategies.  As  illustrated  by  the  AF  taxiway  lighting 
example  in  this  paper,  using  rate,  process,  and  demand  categories  allows  for  a 
straightforward  yet  comprehensive  collection  of  savings  that  result  from  the  implementation 
of  strategic  sourcing. 
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Abstract 

Department  of  Defense  (DoD)  acquisition  requires  information  technology  (IT)  to  undergo  the 
DoD  information  assurance  certification  and  accreditation  process  (DIACAP),  which  makes 
strong  architecture-dependent  assumptions.  Emerging  IT  architectures,  such  as  mobile 
computing  platforms,  invalidate  these  assumptions  and  prevent  the  DoD  from  acquiring 
commercial  technologies  that  are  readily  available  to  adversaries.  To  address  this  problem, 
we  introduce  a  preliminary  framework  in  which  an  application  profile  is  expressed  in  a  formal 
language  and  scaled  with  evolving  architectural  assumptions.  This  profile  aims  to  incorporate 
information  assurance  (IA)  requirements  that  are  commensurate  with  risk  and  scalable  based 
on  an  application’s  changing  external  dependencies.  Information  assurance  risk  levels  that 
account  for  changing  user  identities  and  IA  parameters  (confidentiality,  integrity,  and 
availability)  will  result  from  dynamic  recombination  of  mobile  applications  during  runtime.  The 
language  is  expressed  in  first-order  logic  and  includes  an  evolvable  lexicon  to  describe 
changing  system  configurations.  We  envision  that  software  developers  and  certification 
authorities  can  use  these  formal  profiles  with  an  inference  engine  to  complete  the  DIACAP 
and  maintain  compliance  as  IT  systems  evolve  over  time.  The  framework  has  been  evaluated 
using  existing  DoD  acquisition  and  DIACAP  policy  and  a  case  study  in  a  popular  mobile 
application  ecosystem. 

Introduction 

Network-centric  (net-centric)  warfare  (NCW)  is  the  “generation  of  increased  combat 
power  by  networking  sensors,  decision  makers  and  shooters”  (Alberts,  Garstka,  &  Stein, 
1999).  The  Department  of  Defense  (DoD)  adopted  NCW  as  a  principle  concept  of 
operations  as  early  as  the  late  1990s.  This  adoption  includes  increased  efforts  to  move 
information  from  garrisoned  to  command  posted  and  out  to  “the  edge,”  or  front-line 
combatants,  to  reduce  the  time  from  decision  to  action  and  create  a  more  mobile,  agile,  and 
reactive  force.  During  this  transition,  General  Cartwright  noted  that  NCW  must  decouple  the 
chain-of-information  from  the  chain-of-command  (Onley,  2006;  Carter,  2010)  to  enable  the 
right  people  to  gain  access  to  the  right  information  at  the  right  time.  Unlike  enterprise 
information  systems  in  garrisons  and  command  posts,  computing  at  the  edge  must  be  highly 
dynamic  and  responsive  to  fast-changing  situations.  This  fast-paced  environment  yields 
rapidly  changing  software  requirements,  evolvable  software  architectures,  and  utility 
computing,  which  stress  the  current  DoD  acquisition  system.  The  DoD  acquisition  challenge 
is  that  edge  computing  requires  mobile  applications,  which  are  increasingly  software¬ 
intensive,  developed  on  shorter  timelines,  and  subject  to  different  cyber  security  risks 
(Defense  Science  Board,  2009). 
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The  short  IT  development  timelines  have  led  commercially  available  IT  to  outpace 
the  DoD’s  ability  to  rapidly  acquire  IT  solutions.  This  is  concerning  when  adversaries  can 
acquire  and  deploy  this  technology  worldwide  and  with  few  restrictions.  Mobile  computing 
platforms,  such  as  Apple  iOS  and  Google  Android,  offer  a  stark  contrast  to  traditional 
computing  paradigms  because  they  enable  the  rapid  development  and  deployment  of 
commercial  software  to  handheld  devices,  including  tablets  and  smartphones.  This  software 
integrates  data  from  multiple  sources  and  significantly  reduces  the  time  from  decision  to 
action:  New  “apps”  include  software  to  complete  banking  transactions  by  digitally 
photographing  bank  checks,  to  purchasing  music  based  on  audio  fingerprinting,  to 
integrating  mapping,  routing,  and  directory  services  to  locate  nearby  retailers,  all  within 
seconds.  In  these  examples,  apps  leverage  built-in  devices,  such  as  cameras,  microphones, 
or  geo-location  technologies,  to  create  narrowly  integrated  solutions.  However,  recent  efforts 
to  enable  rapid  DoD  acquisition  has  focused  on  non-software-intensive  systems  (Carter, 
2010;  Wyatt,  2010). 

Mobile  computing  introduces  new  IT  security  risks  within  a  single  IT  system  and 
collectively  across  multiple,  networked  systems.  Unlike  traditional  non-software-intensive 
systems,  IT  security  vulnerabilities  can  compromise  other  systems  on  a  shared  network.  To 
reduce  cyber  security  risk,  the  DoD  Chief  Information  Office  (CIO)  maintains  DoD  Directive 
(DoDD)  8500.1  “Information  Assurance”  and  DoD  Instruction  8500.2  “Information  Assurance 
Implementation,”  which  outline  policy,  responsibilities,  and  procedures  to  integrate  IT 
security  protections  into  DoD  information  systems.  Most  DoD  weapon  systems  subject  to 
DoDD  5000.1  in  the  DoD  Acquisition  System  must  comply  with  these  CIO  policies.  These 
policies  are  primarily  written  for  enterprise  systems,  which  excludes  mobile  computing  as 
envisioned  at  the  edge  in  NCW.  The  challenges  of  modernizing  the  IT  acquisition  policy 
have  been  attributed  to  a  culture  of  buying  large  weapon  systems,  such  as  aircraft  carriers, 
as  opposed  to  incremental  purchases  of  components  that  integrate  into  pervasive,  complex 
systems  (Boessenkool,  2009).  Moreover,  recent  calls  for  modernizing  acquisition  have 
called  for  80%  solutions,  which  is  a  departure  from  complete,  service-centric  solutions  that 
become  outdated  before  they’re  completed  (Gates,  2009).  Mobile  applications  are 
exemplars  of  these  modern  acquisition  challenges. 

The  U.S.  Army  has  been  a  leader  in  the  adoption  of  mobile  applications  in  the  DoD. 

In  March  2010,  the  U.S.  Army  began  the  “Apps  for  the  Army”  challenge,  which  sought  to  test 
a  rapid  acquisition  process  for  software  applications  on  mobile  devices.  The  challenge 
received  53  mobile  applications,  of  which  the  Army  successfully  fielded  25  applications 
through  the  certification  process  (Lopez,  2010).  In  addition,  the  Army  is  actively  engaged  in 
training  mobile  application  developers  in  its  Mobile  Applications  Branch  at  Fort  Gordon 
(Walker,  2011).  Finally,  the  Army  is  taking  steps  to  increase  its  mobile  device  infrastructure: 
After  testing  20-30  smartphones  in  theater,  the  Army  is  now  seeking  to  field  3,500 
smartphones  for  a  single  brigade  (Brewin,  2011).  The  Relevant  ISR  to  the  Edge  (RITE) 
program  recently  completed  testing  and  seeks  to  develop  technologies  to  link  critical  data  to 
soldiers  in  the  field  using  smartphones,  thus  further  pushing  this  paradigm  forward 
(Montalbano,  2011).  These  steps  further  illustrate  the  need  for  adequate  solutions  to  certify 
mobile  applications. 

In  this  paper,  we  propose  a  preliminary  framework  to  model  app  IT-dependencies 
with  the  following  long-term  aims:  (1)  to  reduce  IA  certification  and  accreditation  time  by 
semi-automatically  matching  IA  assumptions  to  application  profiles;  and  (2)  to  extend 
existing  IA  policy  assumptions  to  cover  emerging  mobile  applications  required  in  edge 
computing.  This  paper  is  organized  as  follows.  We  first  review  DoD  IT  acquisition  and  IA 
policy  environment  and  present  policy  gaps  that  inhibit  acquisition  of  mobile  applications; 
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next,  we  briefly  present  our  framework,  application  profile,  and  language  to  address  this 
problem;  finally,  we  discuss  our  evaluation  and  plans  for  future  work  before  concluding  with 
related  work  and  our  discussion. 

DoD  Information  Technology  Policy 

The  DoD  information  technology  (IT)  policy  environment  is  complex  and  distributed 
across  multiple  documents.  The  leading  Department  of  Defense  (DoD)  policies  for  IT 
acquisition  and  information  assurance  (IA)  are  summarized  in  Figure  1.  The  general  DoD 
acquisition  policy  and  responsibilities  are  detailed  in  DoD  Directive  (DoDD)  5000.1,  and  the 
DoD-wide  IA  policy  begins  in  DoDD  8500.1,  which  is  refined  by  IA  controls  contained  in  DoD 
Instruction  8500.2  and  by  the  process  for  performing  IA  certification  and  accreditation, 
described  in  DoDD  8510.1. 


Defense 
Acquisition  System 


Information  Assu  ranee 


DoDD  5000.1 


L 


DoDD  8500.1 


DoDD  5000.2 


u 


DoDI  8500.2 


DoDD  8510.1 


Operation  of  the 
Defense  Acquisition  System 


Information  Assurance 
Implementation 


Defense 

Information  Assu  ranee 
Certification 

and  Accreditation  Process 


Figure  1 .  General  Overview  of  the  DoD  IT  Acquisition  Environment  for 

Information  Assurance 

DoDD  8500.1  governs  information  systems  that  include  mobile  computing  devices, 
such  as  laptops,  handhelds,  and  personal  digital  assistants  (see  DoDD  8500.1,  §  2. 1.2. 7) 
and,  in  particular,  those  devices  that  contain  wired  or  wireless  network  access  to  other 
computing  resources.  This  instruction  covers  four  classes  of  information  system,  as  follows: 

•  Automated  Information  System  (AIS)  Applications,  which  are  products  of  an 
acquisition  program,  such  as  software  applications,  or  a  combination  of 
software  and  hardware,  such  as  workstations,  servers,  and  mobile 
computers; 

•  Enclaves,  which  are  a  collection  of  computing  environments  connected  by 
internal  networks,  under  the  control  of  a  single  authority  and  security  policy; 

•  Outsourced  IT-based  Processes,  which  are  business  processes  supported  by 
private  sector  information  systems;  and 

•  Platform  IT  Interconnections,  which  are  network  access  points  to  computer 
resources  that  are  essential  to  the  mission  in  real-time. 

Figure  2  illustrates  an  example  IS  environment,  consisting  of  two  enclaves,  “A”  and 
“B,”  which  correspond  to  a  garrison  and  command  post,  respectively.  These  environments 
contain  AIS  applications  (square  boxes)  and  outsourced  IT-based  processes  (circles),  some 
of  which  are  DoD  controlled  and  appear  within  the  enclave,  and  others  that  have  shared 
control  and  appear  outside  the  enclave.  Platform  IT  Interconnections  appear  as  solid  black 
arrows:  When  these  connections  exit  an  enclave,  a  demilitarized  zone  (DMZ)  is  assumed  to 
exist  between  the  outgoing  and  incoming  network  traffic.  Mobile  computers,  such  as  laptops 
and  handhelds,  are  a  class  of  AIS  application  that  are  capable  of  moving  across  enclave 
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boundaries;  in  Figure  2,  the  dotted-line  arrows  indicate  movement  of  a  handheld  computer 
from  the  battlefield,  into  a  command  post,  and  later  into  a  garrison.  To  enable  this 
movement,  IA  controls  must  be  in  place  to  avoid  contaminating  these  DoD  controlled 
environments.  For  example,  DoD  8500.1  defines  the  Mission  Assurance  Category  (MAC)  as 
the  level  of  integrity  and  availability  required  by  an  information  system.  Enclaves  always 
assume  the  highest  MAC  of  their  computer  resources  (AIS  applications,  outsourced  IT- 
based  processes,  and  platform  IT  interconnections).  When  an  enclave  connects  to  another 
enclave  that  has  a  lower  MAC  level,  the  enclave  with  the  higher  MAC  level  must  ensure  that 
this  connection  does  not  degrade  the  integrity  and  availability  of  its  computer  resources. 

This  presents  a  particular  challenge  for  mobile  devices  because  they  could  move  across 
enclaves. 


Figure  2.  Example  Information  System  Environment  to  Illustrate  System 

Interactions 

Applications  for  mobile  computers,  specifically  handheld  computers,  have  an 
operational  profile  that  is  situated  among  several  policy  gaps  in  existing  IA  policy.  Under 
DoDD  8500.1,  Designated  Approving  Authorities  (DAAs)  are  responsible  for  certifying  and 
accrediting  these  devices.  Policy  gaps  create  challenging  certification  environments  in  which 
the  DAA  must  assume  insurmountable  risk  for  an  unprecedented  DoD  information  system. 
The  combination  of  changing  users,  changing  applications,  and  changing  locations  is 
characteristic  of  these  devices.  Consequently,  a  solution  is  needed  whereby  configurations 
can  be  reviewed  dynamically  in  the  field  based  on  explicit  IA  assumptions  that  are 
individually  bound  to  the  mobile  device  hardware  and  collections  of  installed  mobile 
applications.  Figure  3  illustrates  a  subset  of  this  complex  policy  environment:  Boxes 
represent  existing  DoD  lA-related  policy;  ovals  represent  IA  controls  from  DoDI  8500.2;  and 
arrows  trace  policy  guidance  from  DoD8500.2  to  IA  controls  and  on  to  other  applicable  DoD 
policies.  Using  our  requirements  specification  language  (Breaux  &  Gordon,  2013),  we 
extracted  a  core  set  of  95  requirements  governing  DoD  IA  responsibilities.  We  now  discuss 
how  these  policies  and  IA  controls  apply  to  mobile  applications,  noting  relevant  shortfalls 
due  to  unique  characteristics  of  this  technology. 
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Figure  3.  Example  DoD  information  Assurance  Policy  Gaps  Affecting  Mobile 

Applications 


Mobile  Code 

Mobile  code  is  defined  by  DoDD  8500.1  to  be  “software  modules  obtained  from 
remote  systems,  transferred  across  a  network,  and  then  downloaded  and  executed  on  local 
systems  without  explicit  installation  or  execution  by  the  recipient.”  DoDI  8500.2  requires  IA 
control  DCMC-1,  which  requires  implementing  mobile  code  policy  in  DoDI  8552.1.  This 
mobile  code  policy  consists  of  approval  decisions  pursuant  to  one  of  three  mobile  code 
categories,  which  are  differentiated  by  (a)  whether  the  mobile  code  is  digitally  signed  by  a 
trusted  certificate;  and  (b)  the  level  of  access  to  operating  system  resources  and  networks 
that  is  granted  to  the  mobile  code.  Mobile  computing  is  enriched  by  mobile  applications 
(code)  that  can  be  downloaded  and  installed  remotely  to  address  emerging  issues. 
However,  the  italicized  phrase  above  in  the  mobile  code  definition  excludes  this  policy  from 
covering  mobile  applications,  despite  that  many  of  the  technical  considerations  (e.g.,  the 
mobile  code  category  differentiators  (a)  and  (b))  are  relevant  to  mobile  applications. 


Mobility 

DoDD  8500.1  defines  mobile  computing  devices  to  include  laptops,  handhelds,  and 
personal  digital  assistants  operating  in  either  wired  or  wireless  mode.  DoDI  8500.2  states 
that  authorized  users  may  “not  relocate  or  change  DoD  information  system  equipment  or  the 
network  connectivity  of  equipment  without  proper  IA  authorization”  (§  5.12.12),  requiring 
advance  IA  authorization  to  move  mobile  computing  devices.  To  employ  compliant  mobile 
computing  devices,  there  is  a  need  to  rapidly  reauthorize  these  devices  as  they  move  within 
and  across  enclaves,  recognizing  that  these  devices  also  contain  mobile  applications,  which 
may  change  over  time  and  thus  change  the  device’s  risk  profile. 

Application  Servers 

The  Defense  Information  System  Agency  (DISA;  2006)  defines  application  server  as 
a  single  computer  that,  in  conjunction  with  other  servers  on  a  network,  provides  an 
application  service  to  a  user  through  a  web  browser.  Application  servers,  such  as  Apache 
Tomcat  and  BEA  Weblogic  Server,  are  “containers”  that  provide  application  infrastructure 
while  server  administrators  can  remotely  deploy  applications  that  are  pre-packaged  as  web 
application  archives  (WARs).  DoDI  8500.2  contains  IA  control  DCCS-2,  which  requires 
compliance  with  available  security  technical  implementation  guides,  or  STIGs.  Only  recently 
were  new  STIGs  developed  to  cover  mobile  device  operating  systems  (iOS,  Android,  or 
Blackberry  OS).  Previously,  the  DISA’s  STIG  governing  application  servers,  which  requires 
responsibility  for  application  server  content  to  be  assumed  by  the  sponsoring  organization  or 
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activity  (DISA,  2006),  were  the  closest  approximation  of  how  mobile  apps  are  installed  on 
mobile  devices,  the  main  difference  being  that  the  STIG  assumes  that  application  servers 
are  fixed  in  an  enterprise  information  system. 

Wireless  Networking 

Wireless  computing  and  networking  capability  is  not  required,  but  it  significantly 
amplifies  mobile  computing  capabilities.  DoDI  8500.2  contains  IA  control  ECWN-1,  which 
requires  that  workstations,  mobile  computing  devices,  and  other  portable  electronic  devices 
comply  with  DoD  wireless  policy.  This  policy  includes  DoDI  8100.2,  §  4.3,  that  states  that 
wireless  devices  may  not  be  operated  in  classified  environments  without  approval  by  the 
DAA  in  consultation  with  CSA  CTTA;  and  §  4.10,  which  requires  a  knowledge  management 
process  to  determine  acceptable  uses  of  wireless  devices  and  appropriate  mitigation 
strategies. 

Virus  Protection 

DoDI  8500.2  contains  IA  controls  ECVP-1,  which  require  virus  protection  for  servers, 
workstations,  and  mobile  computing  devices,  such  as  laptops.  For  general-purpose 
computers,  virus  protection  includes  anti-virus  software  that  recognizes  file  signatures  that 
correspond  to  malicious  code;  this  code  is  downloaded  from  a  remote  computer.  Mobile 
devices  running  restricted  operating  systems,  such  as  Apple’s  iOS  or  Google’s  Android, 
however,  constrain  the  environment  in  which  remote  code  can  be  executed.  In  these 
devices,  mobile  application  infrastructure,  including  pre-approved  applications  in  a  trusted 
app  store,  can  reduce  or  eliminate  exposure  to  malicious  code  for  some  mobile  applications. 

Our  analysis  of  the  Android  2.2  STIG,  Version  1,  Release  1,  yielded  59  requirements 
that  affect  mobile  Android  devices.  Among  these,  40  requirements  target  the  operating 
system,  16  requirements  target  apps,  and  three  requirements  target  external  actions,  such 
as  ensuring  that  mobile  users  respect  the  physical  security  policy  when  using  the 
smartphone  camera.  Requirements  WIR-MOS-AND-006-01  and  WIR-MOS-AND-006-3 
require  approval  from  the  DAA  or  application  control  board  for  all  non-core  apps.  This 
includes  an  inspection  of  the  app  and  risk  analysis.  Although  some  tools  exist  to  conduct 
static  analysis  on  source  code  to  identify  common  vulnerabilities  (e.g.,  buffer  overflows),  to 
our  knowledge  no  tools  exist  to  analyze  mobile  app  requirements  for  the  purpose  of 
identifying  security  risks.  As  a  result,  the  current  guidance  is  inadequate  to  support  the  DAA 
in  evaluating  non-core  apps.  Therefore,  we  now  discuss  our  framework  that  aims  to  begin  to 
address  this  problem. 

Mobile  Application  Framework 

Mobile  devices  that  run  pre-approved  mobile  applications  can  be  viewed  as 
miniature  enclaves,  in  which  the  user  has  the  authority  to  reconfigure  and  recompose  new 
functions  from  multiple  AIS  applications  (mobile  apps)  and  initiate  connections  to  pre¬ 
approved  outsourced  IT  processes  using  platform  IT  interconnections.  Unlike  general- 
purpose  computing  enclaves,  these  applications,  processes,  and  interconnections  operate 
on  pre-defined  data  types  in  a  restricted  computing  environment.  Some  mobile  applications 
leverage  general-purpose  data  types,  such  as  e-mail  clients  or  web  browsers,  but  most  use 
restricted  data  types,  such  as  dates,  locations,  images,  audio,  and  so  forth.  Advances  in 
miniaturization  may  lead  to  mobile  devices  that  run  multiple  computing  environments  in 
parallel,  in  which  each  environment  processes  different  information  classes  with  approved 
guards  for  moving  unclassified  data  into  classified  environments  within  the  same  mobile 
device.  For  example,  recent  work  has  demonstrated  the  ability  to  run  multiple  mobile  OSes 
on  the  same  device  using  virtualization  (Suh  et  al.,  2008).  Finally,  we  envision  that  mobile 
devices  can  be  moved  between  enclaves,  which  changes  the  runtime  assumptions  under 
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which  mobile  applications  are  permitted  to  operate.  Mobile  applications  approved  to  handle 
unclassified  data  cannot  operate  in  classified  physical  and  cyber  environments  without 
approved  guards  to  prevent  the  unauthorized  release  of  classified  data.  Similarly,  classified 
applications  cannot  operate  in  unclassified  environments. 


Figure  4  illustrates  several  challenges  to  accrediting  mobile  applications.  In  Step  1, 
Figure  4,  application  (app)  developers  create  an  application  profile  in  the  EADL  for  their  app 
based  on  the  mobile  application  system  architecture.  In  the  future,  this  profile  may  be 
generated  using  code-level  analysis  to  assist  in  certification  (e.g.,  do  network  connections 
use  OS  SSL  libraries,  or  does  file  I/O  encrypt  data  in  storage?).  In  Step  2,  the  DAA  certifies 
the  app  using  the  application  profile  and  may  accredit  the  app  for  deployment  to  the  mobile 
device  under  this  profile.  The  certification  and  accreditation  includes  a  digital  signature  of 
the  application  profile,  which  the  mobile  device  will  use  to  execute  the  application  only  in 
enclaves  that  conform  to  this  profile.  In  Step  3,  the  signed  app  is  loaded  into  a  DoD  app 
store,  authorized  users  can  download  the  app  to  their  mobile  device. 


estricted 

Server 


nternet 


Legend: 

□  Device  u  AIS  Application  — ►  Platform  1 1  --►Mobility 

Interconnectii 


Figure  4.  Example  Life  Cycle  for  a  Mobile  Application  in  a  Handheld  Device 


In  our  framework,  mobile  applications  are  assigned  to  different  categories  based  on 
their  resource  utilization  profile.  We  envision  the  following  categories:  CAT  1,  2  or  3,  which 
describe  the  level  of  remote  connectivity,  may  be  combined  with  CAT  A  and  B,  which 
describe  the  type  of  local  interactivity.  These  categories  were  validated  based  on  our 
analysis  of  DoD  IA  policy. 

•  CAT  1 :  Stand-alone  apps  are  installed  with  their  complete  data  set,  such  as 
training  manuals,  calculators,  or  dictionaries.  These  apps  do  not  make 
connections  to  remote  servers  or  the  Internet. 

•  CAT  2:  Restricted  apps  periodically  make  connections  to  pre-approved 
servers  only.  These  apps  include  weather  services  and  route-finding 
applications  that  receive  updated  maps  from  pre-approved  sources. 

•  CAT  3:  Unrestricted  apps  may  make  connections  to  remote  servers  that  are 
unsecured  or  not  on  a  pre-approved  list.  This  includes  web  browsers. 
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•  CAT  A:  Apps  may  use  specialized  operating  system  resources,  such  as 
cameras,  microphones,  speakers,  GPS  coordinates,  and  so  forth. 

•  CAT  B:  Apps  may  exchange  pre-defined  data  types  with  other  apps. 

We  examined  descriptions  of  mobile  applications  in  the  context  of  current  initiatives 
in  the  United  States  Army.  Among  the  five  winners  of  the  Apps  for  Army  Challenge,  three 
Apps  could  be  developed  as  CAT  1  apps  (New  Recruit,  Physical  Readiness  Trainer,  and 
Telehealth  Mood  Tracker):  The  first  two  apps  provide  access  to  stable  knowledge  bases  that 
can  be  updated  periodically  in  new,  self-contained  versions  of  the  application;  these  may 
include  hard-coded  web  pages,  training  videos,  and  so  forth  that  reside  locally  on  the  mobile 
device.  The  remaining  two  winners,  Movement  Projection  and  Disaster  Relief  Operations, 
appear  to  be  CAT  2A  apps:  They  rely  on  map-routing  data  that  can  be  acquired  from  an 
approved  source,  such  as  Google  Earth  and  Google  Maps.  If  these  connections  are  not 
secured,  they  could  leak  information  to  intermediaries  who  route  the  data  to  the  map  server, 
which  is  a  CAT  3A  app.  We  envision  that  these  categories  can  be  further  subdivided;  for 
example,  CAT  A  can  be  subdivided  to  distinguish  the  use  of  a  mobile  camera,  versus  the 
use  of  location-based  services  and  accelerometers.  We  further  envision  static  analysis  tools 
that  can  be  developed  to  analyze  the  source  code  of  these  apps  to  automatically  determine 
which  category  the  app  falls  within. 

Mobile  Application  Profile  and  Language  Overview 

The  mobile  application  profile  is  described  by  a  set  of  requirements  to  express  data 
flows  for  a  single  mobile  application.  These  requirements  are  formalized  in  the  description 
logic  (DL)  using  the  semantic  parameterization  method  (Breaux,  Anton,  &  Doyle,  2008). 
Based  on  our  mobile  application  framework,  an  certification  authority  may  want  to  prove 
that,  for  a  given  configuration  (collection  of  mobile  applications),  no  CAT  3A  apps  are 
operating  during  field  operations  that  could  disclose  a  soldier’s  location  to  an  untrusted, 
third-party  server,  or  that  no  communications  exist  between  CAT  2B  and  CAT3B  apps  that 
may  disclose  sensitive  data  to  third  parties. 

To  achieve  this  aim,  we  begin  by  formalizing  a  subset  of  data  requirements  using 
three  Deontic  modalities:  Obligations  describe  what  the  app  is  required  to  do;  prohibitions 
describe  what  the  app  is  prohibited  from  doing;  and  permissions  describe  what  the  app  is 
permitted  to  do.  Formal  requirements  analysis  is  used  to  identify  conflicts  between  what  is 
permitted  and  what  is  prohibited,  noting  that  obligations  imply  permissions  in  Deontic  Logic 
(Horty,  1993).  In  addition,  we  define  a  series  of  roles  based  on  Fillmore’s  (1968)  case 
frames  and  Gruber’s  (1976)  thematic  roles  to  encode  the  actors  engaged  with  the  data,  the 
type  of  data,  and  the  purpose  for  which  the  data  is  used.  At  present,  the  restricted  set  of 
requirements  covers  only  three  specific  actions:  Collection ,  which  is  any  act  to  access, 
assign,  collect,  import,  observe,  or  receive  information  from  another  party,  sensor,  or  device; 
transfer ,  which  is  any  act  to  disclose,  provide,  share,  or  transfer  data  to  another  party;  and 
use,  which  is  any  action  performed  on  the  data  by  the  app  for  a  particular  purpose, 
excluding  collections  and  transfers.  The  set  of  data  requirements  expressed  formally 
constitutes  the  data  flow  aspect  of  the  mobile  application  profile.  App  developers  can 
formalize  these  statements  using  their  knowledge  of  the  app’s  operations,  or  by  using  stated 
natural  language  requirements,  scenarios,  or  privacy  and  security  policies  written  specifically 
for  the  app.  We  now  illustrate  this  formalization  using  an  example  natural  language 
statement  from  a  written  policy. 

Figure  5  portrays  a  statement  that  has  been  encoded  using  the  formalism;  a  more 
complete  description  of  the  formalization  and  an  empirical  case  study  is  described  in  Breaux 
and  Rao’s  (2013)  study.  In  Step  1,  the  analyst  identifies  important  keywords  that  indicate  the 
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action  (e.g.,  import,  enter),  the  modality,  and  the  role  fillers:  the  object  (e.g.,  address  book 
contacts)  is  the  data  type,  and  the  purpose  (e.g.,  locating  contacts)  for  which  the  data  will  be 
used.  Because  the  verbs  import  and  enter  denote  the  movement  of  data  from  the  user  to  the 
app,  these  verbs  indicate  a  collection.  In  Step  2,  the  keywords  are  written  into  a  simple  SQL- 
like  syntax  that  encodes  a  permission,  indicated  by  the  language  operator  P,  followed  by  the 
action  name  and  the  role  fillers.  The  first  role  filler  is  the  object,  followed  by  the  language 
keyword  FOR  that  precedes  the  purpose  for  which  the  action  COLLECT  is  performed.  Using 
DL,  we  can  express  complex  hierarchies  of  data  types,  actor  roles,  and  purposes.  These 
hierarchies  allow  us  to  check  whether  permissions  that  broadly  allow  information  sharing  of 
coarsely  described  information  types  conflict  with  prohibitions  restricting  the  sharing  of 
specific  types.  Conflicts  of  this  type  frequently  arise  due  to  exceptions  in  security  policies. 


Finally,  in  Step  3,  an  automated  tool  parses  the  language  syntax  and  compiles  a  DL 
expression  in  the  Web  Ontology  Language  (OWL).  The  compiled  expression  is  denoted  in 
Figure  5  by  the  two  axioms  for  concept  p8:  The  first  axiom  defines  the  concept  p8  as  a 
collection  action  with  the  appropriate  role  fillers;  the  second  axiom  defines  the  concept  p8  to 
be  a  subclass  of  what  is  permitted.  Using  a  theorem  prover,  we  can  check  these  profiles  for 
internal  consistency  (i.e.,  are  there  any  conflicts  among  the  encoded  data  requirements?). 
We  can  also  check  these  mobile  application  profiles  for  consistency  with  external  properties 
by  expressing  these  properties  in  the  formal  language,  such  as  prohibiting  transfers  to  third 
parties  of  certain  data  types  (e.g.,  e-mail  addresses).  Using  DL,  we  reduced  data 
requirements  conflict  detection  to  DL  satisfiability,  which  is  known  to  be  PsPACE-complete 
for  this  family  of  DL. 


Step  1:  Annotate  policy  text 

—  Modal  phrase  “may”  indicates  an  assumed  permission 

| —  Collect  keyword  p  Datum 

We  may  also  offer  you  the  ability  to  import  your  address  book  contacts 
or  manually  enter  e-mail  addresses  so  that  you  can  (i)  locate  your 
contacts  on  Zynga;  and  (ii)  invite  those  contacts  to  join  you  in  our 
games.  Purposes 

Step  2:  Write  expression  in  specification  language  (P  =  Permission) 

P  COLLECT  addess-book-contacts  FOR  locating-contacts, inviting-contacts 

Step  3:  Compile  language  into  Description  Logic  (OWL) 

p8  =  COLLECT  n  3hasObject. address-book-contacts  n 

BhasPurpose. (locating-contacts  u  inviting-contacts) 
p8  c  Permission 


Figure  5.  Example  Encoding  of  Data  Requirement  Into  Formal  Language 


Case  Study  and  Simulation 

At  present,  the  language  has  been  developed  and  validated  using  information 
privacy  policies  that  describe  privacy  and  security  requirements,  such  as  what  information 
may  be  collected,  used,  and  transferred,  for  what  purposes,  and  by  whom.  These  activities 
architecturally  describe  an  app’s  data  flows  and,  based  on  the  providers  and  recipients  of 
the  data  described  in  these  policies,  these  activities  frame  the  app  in  a  larger  information 
ecosystem  that  consists  of  the  cellular  network  provider,  the  mobile  phone  and  operating 
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system  manufacturers  and  third-party  service  providers  upon  whom  the  app  depends.  Each 
of  these  ecosystem  members  maintain  separate  privacy  policies  that  describe  what  is 
permitted  and  prohibited  with  respect  to  personal  data.  In  addition,  privacy  has  been  viewed 
as  a  “subset  of  security”  because  these  same  information-sharing  policies  also  affect 
confidentiality,  integrity,  and  availability  of  the  system:  Companies  often  aim  to  increase 
information  availability  and  integrity  to  improve  their  services,  while  users  wish  to  increase 
confidentiality  of  personal  information.  Best  security  practices,  such  as  complex  passwords, 
data  encrypted  in  storage  and  in  transfer,  and  so  on,  have  also  been  described  in  these 
policies.  Finally,  these  policies  are  increasingly  required  by  regulators  and  mobile  app 
stores,  such  as  Google  Play  and  iTunes,  which  makes  these  policies  a  pervasive,  publicly 
available,  and  rich  source  of  security  requirements  upon  which  to  develop  a  preliminary 
data-flow  language  for  assessing  our  mobile  security  risk  framework. 

Our  case  study  to  evaluate  the  language’s  formal  semantics  focused  on  the  analysis 
of  three  policies  that  are  linked  in  an  integrated  service  scenario  (Breaux  &  Rao,  2013).  The 
scenario  consists  of  the  Facebook  social  networking  platform,  the  Zynga  game  service,  and 
AOL  advertising  network:  Each  of  these  parties  has  separate  policies  that  govern  the  user’s 
interaction  with  Zynga  online  games,  such  as  Farmville.  Analyzing  these  three  policies 
yielded  374  statements,  of  which  144  statements  were  data  security  requirements  that  were 
expressed  in  the  formal  language.  Among  these,  the  analysis  produced  two  critical  conflicts 
that  we  detected  with  our  automated  theorem  prover:  One  conflict  was  between  the  Zynga 
policy  and  the  Facebook  policy  because  Zynga  reserves  the  right  to  share  information 
obtained  from  Facebook  in  violation  of  Facebook’s  application  developer’s  policy. 

Finally,  we  built  tools  to  parse  and  reason  about  these  policies.  Our  simulation 
studies,  which  are  based  on  these  tools,  show  that  identifying  conflicts  can  be  performed 
within  a  reasonable  amount  of  time  (a  few  minutes)  for  profiles  containing  on  the  order  of 
100  rules.  The  simulation  evaluated  conflict  detection  using  three  different  DL  theorem 
provers:  the  Pellet  OWL2  Reasoner  v2.3.0  developed  by  Clark  and  Parsia,  the  HermIT 
Reasoner  vl.3.4  developed  by  the  Knowledge  Representation  and  Reasoning  Group  at  the 
University  of  Oxford,  and  the  Fact++  Reasoner  vl  .5.2  developed  by  Dmitry  Tsarkov  and  Ian 
Horrocks.  Conflict  detection  for  HermIT  and  FaCT++  was  shown  to  be  linear  and  constant, 
respectively,  with  respect  to  the  number  of  rules.  The  Pellet  reasoner  was  unable  to  scale 
beyond  four  rules  in  our  simulation  due  to  a  design  decision  in  this  version  of  Pellet 
regarding  how  they  handle  a  certain  class  of  DL  expressions.  In  future  work,  we  aim  to 
optimize  conflict  detection  for  larger  policies,  as  needed. 

Future  Work 

In  future  work,  we  plan  to  investigate  three  extensions  to  the  mobile  app  framework 
and  language.  First,  we  aim  to  enhance  our  technical  approach  by  developing  framework 
extensions  to  link  security  specifications  among  multiple,  separate  apps.  These  extensions 
would  allow  an  analyst  to  trace  data  flows  across  multiple  apps  and  thus  check  whether 
security  properties  are  held  across  these  apps  and  their  third-party  services.  We  already 
have  preliminary  evidence  to  demonstrate  this  extension.  Second,  we  aim  to  extend  our 
framework  to  investigate  how  security  policies  change  as  mobile  devices  move  across 
enclaves.  This  requires  establishing  and  comparing  physical  security  policies  for  physical 
locations  with  policies  for  apps  that  interact  with  those  locations,  either  through  user  data 
entry,  location-based  services,  cameras,  or  other  means.  As  we  discussed  with  Figure  2,  we 
aim  to  support  graceful  degradation  to  limit  the  services  that  become  disabled  to  only  those 
that  are  not  trusted  in  high  security  enclaves.  Finally,  we  aim  to  extend  our  formal  language 
with  additional  security  properties  to  describe  how  data  is  properly  stored,  what  attributes 
must  be  true  during  collections  and  transfers,  and  so  forth. 
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In  this  paper,  we  described  an  overview  of  the  approach  and  summarized  our 
evaluation  based  on  policy  analysis  and  a  runtime  simulation.  In  the  future,  we  seek  to  study 
the  use  of  our  framework  in  an  experimental  setting  to  answer  questions,  such  as  the 
following:  How  well  can  app  developers  write  security  specifications  for  their  apps  using  our 
framework,  and  how  well  can  certification  authorities  use  these  specifications  to  assure  the 
app  conforms  to  security  best  practices?  We  imagine  that  new  interfaces  to  our  formal 
methods  would  be  needed  and  that  static  and  dynamic  analysis  tools  could  help  authorities 
verify  that  app  runtimes  conform  to  the  stated  app  security  profile. 

Related  Work 

Related  work  includes  enterprise  architecture  languages,  which  are  used  to  express 
relationships  between  IT  resources  at  a  system-wide  level,  other  work  to  analyze  information 
assurance  policy  in  privacy  and  security,  and  work  to  model  information  assurance 
properties  in  systems.  Enterprise  architecture  (EA)  is  an  informal  concept  consisting  of  four 
layers:  business,  information,  applications,  and  technology.  The  layers  provide  notional 
constructs  for  capturing  the  range  of  personnel  roles,  responsibilities,  assets,  and  functional 
system  requirements.  The  Open  Architecture  Framework  (TOGAF;  The  Open  Group,  2009), 
Department  of  Defense  Architecture  Framework  (DODAF;  DoD  CIO,  2010),  and  Zachman’s 
Framework  (Zachman,  2008)  provide  business  analysts  with  guidelines  and  worksheets  to 
capture  architectural  elements,  but  none  of  these  frameworks  use  formal  languages  to 
enable  model  checking  to  find  inconsistencies  and  conflicts  within  an  architecture. 
Alternatively,  Breaux  and  Powers  (2009)  found  the  Business  Process  Modeling  Notation 
(BPMN),  a  declarative  language  for  describing  business  processes,  to  be  ineffective  to 
express  necessary  temporal  constraints  in  policy  requirements.  Ouyang,  van  der  Aalst,  Ter 
Hoftede,  and  Mendling  (2009)  asserted  that  the  Business  Process  Execution  Language 
(BPEL),  preferred  as  the  candidate  formal  semantics  for  BPMN,  only  works  for  limited 
classes  of  BPMN  models  (Ouyang  et  al. ,  2009). 

Extensive  work  has  been  done  to  model  information  assurance  policies.  Breaux 
developed  an  early  framework  to  extract  privacy  and  security  requirements  from  regulations 
(Breaux,  Vail,  &  Anton,  2006),  which  was  later  validated  in  the  context  of  healthcare  (Breaux 
&  Anton,  2008).  This  work  led  to  the  development  of  a  requirements-specification  language 
to  further  automate  the  encoding  process  (Breaux  &  Gordon,  2013).  More  recently,  this  work 
has  been  formalized  using  DL  to  model-check  requirements  (Breaux  et  al.,  2008)  and  to 
trace  requirements  across  mobile  application  policies  (Breaux  &  Rao,  2013).  Breaux  has 
studied  the  gap  between  security  policy  and  functional  requirements  and  found  a  need  to 
express  both  elements  of  physical  and  cyber  security  architecture  in  the  same  language  to 
reason  about  modern  vulnerabilities  in  distributed  systems  (Breaux  &  Baumer,  2011). 

Discussion  and  Summary 

In  this  paper,  we  presented  a  mobile  application  framework  that  can  be  used  to  map 
existing  DoD  IA  policy  onto  emerging  mobile  devices.  The  aim  of  the  framework  is  to  identify 
opportunities  for  new  methods  and  tools  to  decrease  the  time  required  to  assure  that  mobile 
devices  and  their  applications  conform  to  IA  policies.  Our  approach  consists  of  a  taxonomy 
for  classifying  mobile  apps  based  on  different  security  risks  and  an  application  profile  and 
language  for  describing  data  flows  within  mobile  apps  that  can  be  used  to  check  for  security 
conflicts.  The  profile  describes  what  information  is  collected,  used,  and  transferred,  and  for 
what  purposes;  and  the  language  is  used  to  express  the  profile  formally  and  to  identify 
conflicts  within  and  between  profiles  using  automated  theorem  proving.  In  future  work,  we 
plan  to  extend  the  framework  with  extensions  to  address  potential  policy  conflicts  across 
physical  locations  as  mobile  devices  traverse  different  enclaves.  In  addition,  we  aim  to 
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experimentally  evaluate  this  approach  with  mobile  app  developers  and  certification 

authorities  responsible  for  verifying  that  these  apps  conform  to  relevant  policy. 
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Abstract 

We  present  results  from  our  ongoing  investigation  of  how  best  to  acquire  secure  open 
architecture  (OA)  software  systems.  These  systems  incorporate  software  product  line  (SPL) 
practices  that  include  closed  source  proprietary  software  and  open  source  software  (OSS) 
components,  where  such  components  and  overall  system  configurations  are  subject  to 
different  security  requirements.  The  combination  of  SPLs  and  OSS  components  within  secure 
OA  systems  represents  a  significant  opportunity  for  reducing  the  acquisition  costs  of 
software-intensive  systems.  We  seek  to  make  this  a  simpler,  more  transparent,  and  more 
tractable  process.  Such  a  process  must  be  easy  to  reuse,  adapt,  and  streamline  for  different 
system  application  domains  in  order  to  realize  cost  reductions  and  improve  acquisition 
workforce  capabilities.  Further,  such  a  process  should  be  aligned  with  Better  Buying  Power 
initiatives  addressing  OA  systems,  improved  competition,  defense  affordability,  and 
acquisition  workforce  improvements.  We  identify  different  ways  and  means  for  how  to 
streamline  the  acquisition  process  for  secure  OA  software  systems  through  a  focus  on  doing 
more  with  limited  resources.  Along  the  way,  we  pay  particular  attention  to  revealing  how 
software  licensing  practices  can  affect  cost  in  ways  that  hamper  or  better  the  buying  power  of 
acquisition  programs. 

Introduction 

Our  focus  in  this  effort  is  to  identify  ways  and  means  for  streamlining  the  acquisition 
process  for  secure  open  architecture  (OA)  systems.  These  OA  systems  often  rely  on  the 
integration  of  components  that  are  independently  developed  by  different  software  producers 
and  made  available  as  either  open  source  software  (OSS)  or  proprietary  closed  source 
software  executables.  Program  managers,  acquisition  officers,  and  contract  managers  will 
increasingly  be  called  on  to  review  and  approve  security  measures  employed  during  the 
design,  implementation,  and  deployment  of  OA  systems  (Department  of  Defense  [DoD] 

Open  System  Architecture  [OSA],  2011).  Our  effort  builds  on  both  our  prior  acquisition 
research  (e.g.,  Scacchi  &  Alspaugh,  2008,  2011,  2012a)  and  related  acquisition  research 
efforts  at  the  Program  Executive  Office  (PEO)  Integrated  Warfare  Systems  (IWS;  Guertin  & 
Clements,  2010;  Guertin  &  Womble,  2012;  Womble,  Schmidt,  Arendt,  &  Fain,  2011),  the 
Department  of  the  Navy  (Mactal  &  Spruill,  2012),  and  the  Software  Engineering  Institute 
(SEI)  that  address  software  product  lines  (SPLs;  Bergey  &  Jones,  2010;  Jones  &  Bergey, 
2011;  Northrop  &  Clements,  2007).  It  is  also  influenced  by  related  research  in  the  DoD 
community  addressing  OSS  (Defense  Information  Systems  Agency  [DISA],  2012;  Hissam, 
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Weinstock,  &  Bass,  2010;  Kenyon,  2012;  Martin  &  Lippold,  2011],  component-based 
software  ecosystems  (Scacchi  &  Alspaugh,  2012b;  Reed,  Benito,  Collens,  &  Stein,  2012), 
and  Better  Buying  Power  (BBP)  initiatives  (Defense  Acquisition  University  [DAU],  2012). 

OSS  represents  an  integrated  web  of  people,  processes,  and  organizations, 
including  project  teams  operating  as  virtual  organizations  (Scacchi,  2007,  2009,  2010). 

There  is  a  basic  need  to  understand  how  to  identify  an  optimal  mix  of  OSS  within  OA 
systems  as  products,  production  processes,  practices,  community  activities,  and  multi¬ 
project  (or  multi-organization)  software  ecosystems.  However,  the  relationship  among  OA, 
OSS,  security  requirements,  and  acquisition  is  poorly  understood  [cf.  Scacchi,  2009,  2010; 
Scacchi  &  Alspaugh,  2011,  2012b;  Naegle  &  Petross,  2007],  Subsequently,  from  2007- 
2008,  we  began  by  examining  how  different  OSS  licenses  can  encumber  software  systems 
with  OA,  which  therefore  give  rise  to  new  requirements  for  how  best  to  acquire  software¬ 
intensive  systems  with  OA  and  OSS  elements  (Scacchi  &  Alspaugh,  2008). 

As  a  result  of  our  recent  acquisition  research  efforts,  we  have  been  able  to 
demonstrate  that  it  is  both  possible  and  feasible  to  develop  OA  systems  that  incorporate 
best-of-breed  software  components,  whether  proprietary  or  OSS,  in  ways  that  can  reduce 
the  initial  and  sustaining  acquisition  costs  of  such  systems. 

We  believe  that  such  results  are  applicable  to  both  enterprise  information  systems, 
which  are  widespread  throughout  the  DoD  and  the  U.S.  government,  as  well  as  command 
and  control  (C2;  e.g.,  Reed  et  al. ,  2012;  Scacchi,  Brown,  &  Nies,  2012;  Scacchi  &  Alspaugh, 
2013b)  and  other  defense  systems.  Doing  so,  however,  requires  new  guidance,  and  ideally 
automated  tools,  for  explicitly  modeling  and  analyzing  the  architecture  of  an  OA  system 
during  its  development  and  evolution,  along  with  modeling  and  annotating  the  architecture 
with  software  component  license  rights  and  obligations.  Our  results  thus  demonstrate  a 
major  technological  advance  in  the  acquisition  and  development  of  OA  systems,  as  a 
breakthrough  in  simplifying  software  license  analyses  throughout  the  contracting  activities. 
Creating  similar  advances  for  streamlining  the  acquisition  process,  while  reducing  the  costs 
of  secure  OA  systems,  is  the  next  breakthrough  that  is  needed. 

In  this  paper,  we  describe  ways  and  means  for  how  to  articulate,  tailor,  and 
streamline  the  process  for  how  to  simply  and  transparently  specify  and  assess  OA  system 
security  when  acquiring  different  kinds  of  OA  systems,  and  to  do  so  in  ways  that  highlight 
opportunities  for  cost  reduction  through  system  security  requirements  specification  and  OA 
system  acquisition  process  streamlining.  We  provide  examples  of  complex  software 
elements  that  are  applicable  to  many  kinds  of  software-intensive  systems  within  the  DoD  as 
well  as  within  other  government  agencies  and  industrial  firms.  But  we  start  in  the  next 
section  by  reiterating  BBP  principles  and  initiatives  that  guide  this  research  by  focusing  on 
how  to  promote  competition  in  the  acquisition  and  development  of  secure  OA  systems. 

Open  Architecture  and  Better  Buying  Power 

BBP  (see  http://bbp.dau.mil/)  is  part  of  the  DoD’s  mandate  to  do  more  without  more 
by  implementing  best  practices  in  acquisition.  BBP  identifies  seven  areas  of  focus  that  group 
a  larger  set  of  36  initiatives  that  offer  the  potential  to  restore  affordability  in  defense 
procurement  and  improve  defense  industry  productivity.  One  of  the  seven  areas  focuses  on 
promoting  competition,  and  this  area  includes  an  initiative  to  “enforce  open  system 
architectures  and  effectively  manage  technical  data  rights”  (DAU,  2012).  Technical  data 
rights  pertain  to  two  categories  of  intellectual  property  (IP):  they  refer  to  the  government’s 
rights  to  (a)  technical  data  (TD;  e.g.,  product  design  data,  computer  databases,  computer 
software  documentation)  and  (b)  computer  software  (CS;  e.g.,  source  code,  executable 
code,  design  details,  processes,  and  related  materials).  These  rights  are  realized  through  IP 
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licenses  provided  by  system  product  or  service  providers  (e.g.,  software  producers)  to  the 
government  customer,  so  long  as  the  customer  fulfills  the  obligations  stipulated  in  the 
license  agreement  (e.g.,  to  indicate  how  many  software  users  are  authorized  to  use  the 
licensed  product  or  service  according  to  a  fee  paid).  As  already  noted,  our  acquisition 
research  has  focused  on  issues  addressing  OA  systems  and  IP  licenses  since  2008 
(Scacchi  &  Alspaugh,  2008). 

OA  software  systems  offer  the  potential  to  improve  acquisition  by  providing  new 
ways  and  means  to  acquire,  develop,  deploy,  and  sustain  software-intensive  systems. 

These  new  ways  and  means  in  turn  may  transform  how  the  DoD  acquires  complex  systems 
by  moving  away  from  long-duration,  proprietary  (closed)  system  architecture,  and  the 
difficult-to-control  cost  of  system  development  efforts,  towards  systems  that  may  be  more 
rapidly  assembled/integrated  in  an  OA  manner  with  more  transparent  costs.  Such  a 
transformation  may  in  turn  reduce  vendor  lock-ins  that  oftentimes  are  associated  with  rising 
costs  to  sustained  deployed  systems  that  are  inaccessible  to  competing  vendors.  So  closed 
architecture  legacy  systems  are  often  subject  to  IP  licenses  whose  consequence  is  to 
reduce  competition  while  increasing  system  sustainability  costs.  Our  research  on  OA 
systems  dating  many  years  back  (Scacchi  &  Alspaugh,  2008)  has  consistently  been  aligned 
with  efforts  for  improving  competition  in  software  system  development  and  evolution  through 
an  investigation  of  innovative  ways  and  means  to  acquire/develop  component-based  OA 
software  systems  that  are  subject  to  diverse,  heterogeneous  IP  licenses  (Alspaugh, 

Scacchi,  &  Asuncion,  2010).  But  there  is  more  to  do  to  improve  competition  and  defense 
affordability  while  effectively  managing  technical  data  rights  when  addressing  the  acquisition 
of  secure  OA  systems.  In  particular,  this  includes  understanding  that  the  processes  for 
acquiring  such  systems  are  facilitated  or  constrained  in  light  of  overall  BBP  guidance  and 
best  practices  as  well  as  how  best  to  improve  and  streamline  these  processes.  These  topics 
are  our  focus  in  the  remainder  of  this  paper. 

How  Better  Buying  Power  Impacts  the  Processes  for  Acquiring  OA  Systems 

The  move  to  OA  systems  represents  a  transition  from  the  acquisition  of  monolithic 
systems  to  the  acquisition  of  reusable  system  components  that  can  be  integrated  to  realize 
different  configurations  of  a  software  product  family  for  a  specific  application  domain 
(Bergey  &  Jones,  2010;  Guertin  &  Clements,  2010;  Jones  &  Bergey,  201 1 ;  Reed  et  al., 

2012;  Scacchi  &  Alspaugh,  2012b;  Northrop  &  Clements,  2007;  Womble  et  al.,  2011).  These 
components  are  acquired  within  a  software  ecosystem  that  is  evolving  towards  component 
provisioning  within  open  repositories,  where  components  from  different  producers  are 
available  for  selection,  evaluation,  and  system  integration  (Guertin  &  Womble,  2012;  Martin 
&  Lippold,  201 1 ;  Reed  et  al.,  2012;  Scacchi,  2007;  Scacchi  &  Alspaugh,  2012a,  2013b). 
Figure  1  provides  a  graphic  view  of  how  such  an  ecosystem  spans  from  a  sample  of 
software  producers  and  components  through  system  integrators  to  software 
consumers/users. 
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Figure  1.  A  Sample  Software  Ecosystem  of  Producers,  Components, 
Integrators,  Alternative  OA  Systems,  and  Consumers/Users 

(Scacchi  &  Alspaugh,  2012b) 

Figure  2  provides  a  view  of  a  sample  of  lightweight  software  components  (“widgets” 
targeted  for  software  developers  or  integrators  in  this  example)  for  download  and  installation 
within  a  Web  browser.  These  widgets,  made  by  different  producers,  are  available  for 
acquisition  from  Google’s  Chrome  Web  Store. 
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Figure  2.  A  Sample  View  of  Lightweight  Software  Components  (“Widgets”)  That  Can  Be 
Readily  Acquired  for  Evaluation  or  Integration  From  Google’s  Chrome  Web 

Store 


Such  an  online  store  serves  as  a  marketplace  that  provides  access  to  ready-to-run, 
closed  source  software  executables  from  within  an  online  software  repository  that  can  be 
navigated  using  the  menu  on  the  left  side,  browsed  by  scrolling,  or  accessed  by  entry  of  a 
search  term/phrase  in  the  upper-left  corner  (see  Figure  2). 

Software  components  in  an  online  marketplace  like  this  are  rated  or  recommended 
by  other  consumers,  but  the  IP  licenses  for  the  TD  and  CS  are  hidden  away  with  each 
component  and  may  be  challenging  to  locate  prior  to  installation.  Google  Play  for  Android 
Apps  and  the  Apple  App  Store  also  offer  software  (widget)  components  for  their  respective 
computing  platforms  (Android  and  iPhone  smartphones,  or  Nexus  and  iPad  mobile  tablet 
computers). 

Figure  3  provides  a  view  of  a  different  online  repository  that  exclusively  features  OSS 
components  found  at  SourceForge.net  (similar  to  Forge.mil  [DISA,  2012;  Martin  &  Lippold, 
201 1]),  where  the  IP  licenses  for  each  software  component  are  prominently  displayed  when 
one  selects  to  look  more  closely  into  the  details  and  development  status  of  a  component  of 
interest.  In  contrast  to  the  Web-browser-specific  software  widgets  available  at  the  Chrome 
Web  Store,  the  OSS  components  at  SourceForge.net  represent  more  substantial, 
production-oriented  software  tools  or  utilities  that  can  operate  as  stand-alone  application 
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programs.  Forge.mil  may  be  envisioned  to  provide  support  for  accessing  pre-tested  and 
certified  software  components,  whether  lightweight  widgets  or  more  substantial  application 
systems,  in  OSS  code  and  ready-to-run  executable  forms  with  technical  data  rights 
designed  for  government  purposes.  Thus  overall,  what  we  see  is  that  if  we  want  to  improve 
competition  through  the  acquisition  of  component-based  software  systems,  our  choice  of 
which  online  repository  or  marketplace  to  use  leads  to  different  kinds  of  software 
components  with  different  IP  license  schemes. 
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Figure  3.  Sample  of  OSS  Security/Utility  Components  Found  at  SourceForge.net 


Next,  we  encounter  challenges  in  the  development  of  integrated  OA  systems  that  are 
configured  from  different  software  components.  Figure  4  provides  a  visual  representation 
showing  that  different  software  producers  can  develop  different  kinds  of  software 
components  (small,  medium,  or  large  size/capability),  which  system  integrators  can  select 
from  in  order  to  create  an  OA  system  product  line  of  alternative  component  configurations. 
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Figure  4.  A  Component-Based  Software  Ecosystem  That  Configures  a  Product  Line  of 
Four  Alternative  System  Configurations,  Conforming  to  an  OA  System  Design 

in  Figure  5 

Figure  5  shows  a  simple  OA  system  design  that  accommodates  alternative  software 
components  as  applications  or  infrastructure  elements  that  may  be  subject  to  OSS  or 
proprietary  licenses.  The  applications  (“apps”)  may  include  small,  proprietary,  and 
lightweight  browser  widgets  or  large  components  like  OSS-based  Web  browsers.  The 
infrastructure  software,  which  is  assumed  to  serve  as  an  independent  foundation  for 
application  software,  can  include  proprietary  or  OSS  components  like  database 
management  systems  (or  network  file  systems  or  other  online  repositories)  and  computer 
operating  systems.  Figure  6  displays  the  selection  of  one  set  of  conforming  software 
components  selected  from  the  software  ecosystem  in  Figure  4  that  also  conforms  to  the  OA 
system  design  in  Figure  5. 
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Figure  5.  A  Simple  OA  System  Design  That  Accommodates  Software  Components  as 
Applications  or  Infrastructure  Elements,  Shown  in  Figure  4 
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Figure  6. 


A  Selection  of  Software  Components  From  the  Ecosystem  in  Figure  4 
Conforming  to  the  OA  System  Design  in  Figure  5 


Lightweight  software  widgets  are  developed  using  domain-specific  scripting 
languages,  like  JavaScript  or  PHP,  which  are  designed  to  operate  with  popular  Web 
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browsers  or  browser-based  integrated  system  environments.  These  widgets  commonly 
represent  small  programs  that  are  often  produced  with  limited  resources  on  short  time 
frames  and  sometimes  constitute  only  hundreds  of  lines  of  scripting  source  code.  More 
complex  integrated  capabilities  can  be  constructed  by  integrating  a  set  of  selected  widgets 
using  additional  scripting  code  via  integration  techniques  that  produce  inter-application 
“mashups.”  Consequently,  there  is  substantial  competition  in  the  widget/app  marketplace. 
However,  these  lightweight  software  components  often  have  short-term  life  cycles,  and  few 
updates  before  their  demise. 

At  present,  lightweight  software  components  tend  not  to  be  sustained  for  periods 
beyond  their  early  availability,  widespread  adoption,  and  deployment.  Their  life  cycle  may  be 
measured  in  months,  rather  than  years  (or  decades).  Consequently,  these  lightweight 
components  are  effectively  designed  to  be  disposable,  low-cost  software — acquire  it,  then 
use  it  until  something  better  is  available,  then  repeat.  This  means  that  it  may  be  easier  for 
producers  of  such  components  to  develop  new  components  with  new(er)  capabilities, 
technologies,  or  remote  services,  rather  than  trying  to  sustain  the  short-lived  legacy  code.  In 
this  regard,  producing  new  components  may  be  less  costly  than  maintaining  legacy 
components  that  depend  on  technologies  or  services  that  may  no  longer  be  available  or 
viable.  Lightweight  software  components  with  short  life  cycles  in  this  regard  may  improve 
competition,  overall  system  adaptability,  and  affordability  while  reducing  vendor  lock-in  to 
costly  legacy  software.  Updated  versions  of  such  components  may  be  provided  to  repair  or 
replace  problematic  implementations,  but  they  may  also  appear  simply  as  an  inducement  to 
maintain  use  of  the  component  until  an  extended  (e.g.,  “pro”)  version  becomes  available  for 
acquisition.  Finally,  the  globally  dominant  online  app  stores  like  those  operated  by  Apple, 
Blackberry,  Google,  Microsoft,  and  others  tend  to  primarily/exclusively  distribute  small, 
lightweight  software  components  as  proprietary  closed  source  executables  on  a  per-user 
basis,  and  with  IP  licenses  that  prohibit  open  access,  reuse,  modification,  and  redistribution. 
But  these  choices  are  determined  by  the  business  models  of  the  online  repository/store 
operators,  rather  than  on  some  critical  technological  dependency  or  constraint.  So  new 
software  products  like  lightweight  components  from  online  repositories/stores  will  likely 
require  more  agile  acquisition  processes,  contracting  practices,  and  replacement/upgrade 
and  IP  license  management  regimes. 

In  contrast,  the  Web  browsers  in  which  these  widgets  run  are  themselves  substantial 
multi-million  source  lines  of  code  software  components  that  are  often  integrated  into  larger 
software-intensive  defense  systems,  like  the  C2RPC  experimentation  platform  (Garcia, 

2010;  Gizzi,  2011).  These  browsers  and  other  integrated  software  packages  are  tested  and 
deployed  on  global  scales,  which  in  turn  helps  to  insure  their  viability,  sustainability,  and 
quality  within  a  highly  competitive  software  product  ecosystem.  Their  availability  as  either 
proprietary  or  OSS  forms  indicates  that  there  is  active,  ongoing  competition  among  their 
producers.  In  addition,  these  OSS  browsers  and  other  integrated  software  packages  based 
on  open  standards  (e.g.,  OpenOffice,  LibreOffice)  mean  that  commonly  used,  large-scale 
software  applications  and  software  infrastructure  systems  are  available  with  IP  licenses  that 
offer  lower  acquisition  costs  and  improved  competition,  as  well  as  improved  defense 
affordability  options. 

OSS  components  found  at  SourceForge.net  or  Forge.mil  are  typically  somewhere  in 
between  in  size,  complexity,  and  functional  capability  of  lightweight  widgets  and  large 
integrated  software  packages.  However,  there  is  no  requirement  imposed  in  OSS 
repositories  about  what  size,  complexity,  or  capability  components  can  be  made  available. 
So  many  OSS  components  range  in  size  from  thousands  to  hundreds  of  thousands  of 
source  lines  of  code,  and  they  vary  in  terms  of  their  quality  and  sustainability.  OSS 
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components  from  online  repositories  like  SourceForge.net  are  generally  available  for  free  or 
for  a  low  cost  and  may  or  may  not  be  designed  around  open  standards.  Many  OSS-based 
applications  do  not  rely  on  any  standards,  while  much  OSS-based  infrastructure  software 
relies  on  either  open  industry  standards  or  de  facto  standards  grounded  in 
proprietary/legacy  systems  (sometimes  referred  to  as  “workalike”  or  functionally  similar 
[Scacchi  &  Alspaugh,  2012b]  systems).  In  contrast,  the  DoD  is  seeking  to  make  sure  that  its 
online  OSS  repositories  like  Forge.mil  (or  others)  will  only  host  components  that  are  pre¬ 
tested  and  certified  as  compliant  with  relevant  standards,  quality/reliability  indicators,  and 
security  policies  relevant  to  their  problem  domain  (DISA,  2012;  Kenyon,  2012;  Reed  et  al. , 
2012], 

Software  components  and  online  component  repositories/stores  offer  the  potential  to 
transform  the  ways  and  means  for  acquiring  and  developing  component-based  OA  systems. 
But  at  present,  the  size,  functional  complexity,  quality,  extensibility,  and  sustainability  of 
different  software  components  vary  in  part  based  on  the  repository/store  from  which  they  are 
acquired.  Although  components  that  can  be  integrated  within  a  secure  OA  system  offer  the 
potential  to  increase  competition,  the  acquisition  processes  need  to  be  updated  and  the 
acquisition  workforce  newly  trained  in  these  new  ways  and  means  in  order  to  maximize  the 
likelihood  for  BBP  initiatives  addressing  OA  systems. 

How  Best  to  Improve  and  Streamline  Acquisition  Processes  for  Secure  Open 
Architecture  Systems 

The  transition  to  the  development,  deployment,  and  sustainment  of  software¬ 
intensive  systems  based  on  an  OA  means  that  new  or  revised  acquisition  processes  may  be 
needed.  In  particular,  we  believe  that  such  advances  call  for  (a)  the  adoption  of  open 
business  models  within  the  DoD  and  its  industry  partners,  (b)  open  source  approaches  to 
creating  Web-based  acquisition  processes  (Scacchi,  2001)  that  specifically  address  BBP 
initiatives,  and  (c)  employing  techniques  for  streamlining  these  processes  (Choi  &  Scacchi, 
2001;  Nissen,  1998;  Scacchi  &  Noll,  1997;  Scacchi,  2001)  for  secure  OA  systems.  Each  is 
described  in  turn  in  this  section. 

Encouraging  the  Adoption  of  Open  (Source)  Business  Models 

One  goal  of  BBP  initiatives  is  to  reduce  costs  by  improving  competition.  Such  a 
situation  may  be  disconcerting  to  legacy  software  producers  who  are  long  experienced  with 
the  long-term  development  of  proprietary,  large-scale  software  systems  with  closed 
architectures  that  are  subject  to  traditional,  cumbersome,  and  costly  software  product 
licenses  and  license  management  regimes  (Anderson,  2012;  Konary,  2009).  A  move 
towards  the  agile  and  adaptive  development  of  secure  OA  systems  based  on  software 
components — that  can  be  developed/integrated  more  rapidly  and  at  a  lower  cost  with  more 
favorable  IP  licenses — represents  a  new  acquisition  strategy  (Reed  et  al.,  2012;  Scacchi  & 
Alspaugh,  2013b).  This  suggests  the  need  to  incentivize  software  producers  and  system 
integrators  so  as  to  insure  their  ability  to  effectively  produce  both  proprietary  and  OSS 
components  that  are  economically  viable  yet  cost  effective  to  the  government  over  the  life  of 
such  systems.  The  overall  BBP  mandate  recognizes  this  situation  but  does  not  specify  the 
means  for  how  best  to  accomplish  it.  We  believe  that  one  promising  candidate  is  for  defense 
enterprises  and  program  offices  to  adopt  new  open  business  models. 

The  business  models  that  we  have  in  mind  should  be  rendered  in  an  open  source 
format.  Such  models  should  be  computer  processable  (i.e.,  amenable  to  automated 
enactment  support)  and  transparent  to  participants  in  the  acquisition  workforce  (e.g., 
available  through  Web-based  application  systems  [Scacchi,  2001;  Scacchi  &  Noll,  1997]). 
They  should  similarly  be  open  to  participants  in  software  producer,  system  integrator,  and 
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system  user  enterprises.  These  models  should  incorporate  a  product  line  of 
common/reusable  open  system  architectures  that  can  integrate  functionally  similar  software 
components  in  order  to  realize  domain-specific  system  solutions  (e.g.,  for  domains  like  C2, 
weapon  systems,  or  enterprise  computing;  Bergey  &  Jones,  2010;  Guertin  &  Clements, 

2010;  Jones  &  Bergey,  2011;  Reed  et  al.,  2012;  Scacchi  &  Alspaugh,  2012b;  Northrop  & 
Clements,  2007;  Womble  et  al.,  2011).  These  business  models  should  incorporate  Web- 
based  computational  models  of  acquisition  processes  (Nissen,  1998;  Scacchi,  2001; 

Scacchi  &  Noll,  1997)  that  manage  the  system  development  and  support  processes  that 
surround  the  OA  product  line  system  models.  Finally,  these  business  models  should 
highlight  which  acquisition  or  system  development  processes,  or  OA  system  features, 
require  attention  to  IP  licenses. 

Prior  research  has  demonstrated  that  significant  cost  reductions  and  process 
streamlining  are  possible  when  open  source  business  process  models  are  utilized  (Choi  & 
Scacchi,  2001;  Nissen,  1998;  Scacchi  &  Noll,  1997;  Scacchi,  2001).  These  kinds  of  models 
can  be  subjected  to  performance  measurements  across  multiple  acquisition  process 
enactments,  continuous  improvement,  and  process  redesign  by  the  acquisition  workforce 
(Scacchi,  2001).  Now  we  propose  to  enhance  and  extend  their  value  through  the 
incorporation  of  OA  system  models.  While  demonstrating  such  a  capability  is  beyond  the 
scope  of  this  study,  prior  research  results  suggest  the  plausibility  of  such  an  approach.  So 
future  acquisition  research  targeting  BBP  may  be  directed  to  the  creation  of  open  business 
models  that  can  be  openly  accessed,  reused,  modified,  and  redistributed  where  appropriate. 

Open  Source  Models  of  Acquisition  Processes 

As  noted,  prior  research  has  demonstrated  the  value  and  real  payoffs  of  Web-based 
computational  models  for  defense  acquisition  processes  (Choi  &  Scacchi,  2001;  Nissen, 
1998;  Scacchi  &  Noll,  1997;  Scacchi,  2001).  However,  many  technological  advances, 
organizational  transformations,  and  shifting  defense  priorities  have  occurred  since  these 
results  were  first  demonstrated  and  deployed  years  ago.  Our  own  studies  on  the  design  of 
secure  OA  system  product  lines  are  an  example  of  technological  advances  not  addressed  in 
our  earlier  process  models.  But  without  explicit,  open  source  process  models  that  can  be 
enacted  through  Web-based  user  interfaces  (i.e.,  Web  browsers  accessing  remote 
application  services  while  tracking  process  enactment  progress  and  performance 
parameters),  the  ability  to  realize  their  benefits  (like  process  streamlining  and  cost  reduction) 
is  elusive  and  difficult  to  manifest.  Among  the  reasons  for  why  this  is  so  includes  overcoming 
gaps  for  how  best  to  (a)  monitor  and  measure  acquisition  process  performance  without 
automated  enactment  support;  (b)  redesign  legacy  processes  to  better  accommodate 
technical  advances  and  to  remove  ineffective  bureaucratic  procedures,  or  that  transform 
acquisition  processes  in  ways  that  do  more  with  less  while  also  empowering  the  acquisition 
workforce;  (c)  design  new  acquisition  processes  like  those  for  acquiring  secure,  component- 
based  OA  software  systems  subject  to  multiple  IP  licenses;  and  (d)  accommodate  software 
IP  licenses  and  license  management  regimes  as  acquisition  process  cost  elements.  To 
better  understand  what  gaps  exist  in  these  four  areas,  we  now  describe  techniques  for 
streamlining  the  acquisition  processes  for  secure  OA  systems. 

Techniques  for  Streamlining  Acquisition  Processes  for  Secure  Open  Architecture 
Systems 

A  goal  of  this  paper  is  to  identify  ways  and  means  for  streamlining  acquisition 
processes  for  secure  OA  systems.  In  particular,  we  focus  on  four  kinds  of  techniques  that 
can  be  used  to  streamline  such  processes  in  ways  that  are  responsive  to  the  BBP  initiative 
for  open  system  architectures  subject  to  complex  IP  licenses.  These  techniques  are 
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illustrative  rather  than  exhaustive  since  other  kinds  of  techniques  in  other  areas  are  also 
expected  to  exist  and  be  available  for  practice  by  the  acquisition  workforce. 

Process  Measurement  and  Assessment 

The  most  direct  way  to  determine  the  efficiency  and  effectiveness  of  acquisition 
processes  is  by  measuring  their  structural  attributes.  Such  attributes  indicate  things  such  as 
(a)  the  length  of  the  longest  path  of  process  steps/actions  (process  length);  (b)  the  number 
of  distinct  process  paths  (process  width);  (c)  the  number  of  sub-process  levels  (process 
depth);  (d)  the  total  number  of  process  steps  (process  size);  and  (e)  the  process  size  divided 
by  process  length  (process  parallelism)  as  well  as  others  metrics  (Nissen,  1998).  But  without 
an  explicit  graph-based  model  of  acquisition  processes,  such  measurements  are  impractical 
or  implausible.  Nonetheless,  such  metrics  are  a  key  for  where  to  look  for  process 
improvement  or  process  redesign  opportunities.  One  might  also  recognize  that  some 
acquisition  processes  are  underspecified — for  example,  by  not  explicitly  accounting  for 
where  software  licenses  are  negotiated  or  license  trade-off  analysis  is  done.  Similarly, 
because  OA  systems  may  include  software  components  subject  to  different  licenses 
(Alspaugh  et  al. ,  2010),  how  are  component-component  license  interactions  assessed  or 
analyzed,  if  at  all?  If  acquisition  processes  do  not  explicitly  account  for  new  acquisition  or 
license  management  activities  that  emerge  due  to  advances  in  OA  system  development, 
then  such  processes  are  underspecified,  which  means  their  costs  are  hidden  and  difficult  to 
control/minimize.  Thus,  if  the  goal  of  BBP  is  to  help  improve  the  affordability  of  OA  systems 
within  the  DoD,  then  we  need  to  be  able  to  systematically  model,  measure,  and  assess  our 
acquisition  processes  (Scacchi,  2001).  Similarly,  we  need  to  better  understand  how  to 
measure  and  assess  open  business  models  for  use  within  the  DoD  and  its  industry  partners 
to  incentivize  and  continuously  improve  competition  and  defense  affordability 

Process  Redesign  and  Evolution 

Once  we  have  the  ability  to  measure  and  assess  current/emerging  acquisition 
processes  for  secure  component-based  OA  systems,  we  can  then  begin  to  analyze  (or 
simulate)  them  in  ways  that  reveal  process  redesign  opportunities  and  transformation 
heuristics  (Choi  &  Scacchi,  2001;  Nissen,  1998;  Scacchi  &  Noll,  1997;  Scacchi,  2001). 
Among  the  acquisition  process  pathologies  we  seek  to  identify  are  those  where  measured 
processes  reveal  sub-processes  with  low  effectiveness  (indicating  high  levels  of  iterative 
rework),  low  efficiency  (indicating  slow  or  bureaucratically  cumbersome  process  steps  that 
add  marginal  value  to  process  completion),  and  problematic  sub-processes  (indicating 
underspecified  process  steps,  steps  that  generate  processing  delays  due  to  missing  and/or 
incorrect  acquisition  data,  or  inappropriate  automated  process  enactment  support).  For 
example,  current  processes  that  assume  the  long-term  acquisition  of  monolithic  software 
systems  with  proprietary  components  integrated  within  a  closed  architecture  are  likely  not 
well  suited  to  address  the  challenges  for  acquiring  secure  OA  systems  that  integrate 
software  components  from  different  online  repositories.  We  also  place  our  acquisition 
workforce  at  a  disadvantage  if  we  do  not  empower  them  with  the  ability  to  measure,  assess, 
and  adaptively  redesign  their  processes  as  technological  advances  like  component-based 
OA  systems  are  to  be  acquired.  New  software  component  technologies  and  software 
ecosystem  niches  (Scacchi  &  Alspaugh,  2012a)  are  also  emerging,  which  necessitate  new 
continuous  development  processes  and  new  license  management  practices  and  thus  the 
redesign/evolution  of  acquisition  processes  (Scacchi  &  Alspaugh,  2013a;  Scacci  et  al., 

2012).  These  examples  all  point  to  new  opportunities  to  redesign,  evolve,  or  otherwise 
transform  existing  acquisition  processes  to  better  fit  the  challenges  posed  by  the 
development,  deployment,  and  support  of  secure,  component-based  OA  systems.  Finally, 
we  can  empower  the  acquisition  workforce  to  realize  continuously  improved  acquisition 
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processes  if  we  can  provide  them  with  the  training  and  resources  for  modeling,  analyzing, 
and  redesigning  their  acquisition  processes  in  ways  that  utilize  Web-based  automated 
process  enactment  systems,  which  also  allow  them  to  try  out  and  walk  through  alternative 
process  redesigns  before  committing  to  their  use  in  daily  operations. 

Design  New  Acquisition  Processes 

Across  the  DoD  community,  there  are  many  variations  in  practice  for  how  to  specify 
and  model  the  architecture  of  a  software-intensive  system.  Some  practices  focus  attention 
primarily  on  identification  of  major  components  or  abstract  layers  while  minimizing  (or 
ignoring)  attention  to  interfaces  and  interconnections,  which  are  more  challenging  to  identify 
and  manage.  However,  the  BBP  initiative  for  OA  systems  points  to  the  need  for  managing 
explicit  interface  specifications  that  identify  and  reinforce  the  use  of  standard  interfaces 
(DAU,  2012).  Without  such  interface  and  interconnection  specifications,  it  is  not  possible  to 
determine  the  scope  or  potential  conflicts/matches  between  the  IP  licenses  (and  thus  TD 
rights)  for  the  overall  system  architecture.  In  contrast,  we  have  demonstrated  in  our  prior 
research  that  component-based  OA  systems  become  tractable  and  evolvable  from  IP 
license  management  and  security  perspectives  when  the  system  architecture  of 
components,  connectors,  and  interfaces  are  explicitly  modeled  (Alspaugh  et  al.,  2010; 
Scacchi  &  Alspaugh,  2011,  2012a,  2012b,  2013b).  The  use  of  standard  interfaces  further 
allows  for  simpler  renderings  of  OA  system  structure,  and  thus  simplifies  license  analysis. 
Further,  once  interfaces  and  interconnections  become  explicit,  software  component 
producers,  system  integrators,  and/or  system  consumers  can  determine/negotiate  which 
interfaces  should  be  standardized  in  order  to  improve  competition  and  affordability.  These 
standards  may  then  define  acceptable  data  types,  relationships  between  data  types,  data 
attribute  value  ranges,  and  exceptional  data  values  in  ways  that  are  open,  sharable,  and 
reusable  as  well  as  extensible  when  appropriate.  Such  improvements  become  possible  by 
enabling  an  agile,  adaptive  ecosystem  for  software  components  of  different  size  and 
capability  relative  to  OA  system  product  lines  for  different  application  domains  (Reed  et  al., 
2012;  Scacchi  &  Alspaugh,  2012a,  2013b).  Therefore,  another  important  technique  for 
streamlining  the  acquisition  of  secure,  component-based  OA  systems,  in  line  with  BBP 
initiatives,  is  to  provide  the  acquisition  workforce  with  the  resources  and  automated  support 
to  design  and  computationally  enact  new  acquisition  processes  (i.e.,  explicitly  modeled 
processes;  Choi  &  Scacchi,  2001;  Nissen,  1998;  Scacchi  &  Noll,  1997;  Scacchi,  2001), 
where  the  processes  are  open,  agile,  and  adaptive.  Such  modeled  processes  may  also  then 
be  shared,  reused,  continuously  improved,  and  redistributed  across  the  ecosystem  of 
defense  enterprises  and  program  offices. 

Cost  Management  as  a  Process  Design  Element 

Part  of  the  promise  of  the  move  to  OA  systems  stems  from  their  perceived  potential 
to  reduce  acquisition  life  cycle  costs,  improve  competition,  and  improve  defense  affordability 
(DAU,  2012).  But  where  and  how  are  the  associated  cost  factors  or  cost  drivers  for  OA 
systems  identified,  tracked,  and  managed?  After  all,  if  we  do  not  know  where  the  cost 
factors  are,  or  what  activities,  conditions,  or  events  drive  OA  system  acquisition  costs,  then 
we  cannot  effectively  control  such  costs  nor  make  well-informed  system  capability/cost 
trade-offs.  For  example,  people  who  manage  the  acquisition  of  large-scale  software  systems 
within  various  defense  enterprises  are  familiar  with  the  many  types  of  end-user  license 
agreements  for  proprietary,  closed  source  software  systems  (Anderson,  2012).  In  contrast, 
these  people  may  not  know  how  best  to  manage  the  acquisition  of  OA  systems  whose 
software  components  are  jointly  subject  to  different  OSS  or  proprietary  licenses. 

The  acquisition  workforce  have  also  learned  in  practice  that  software  IP  licenses  are 
subject  to  change  over  time.  However,  one  consequence  is  that  long-lived  or  widely  used 
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software  systems  become  more  costly  and  much  less  amenable  to  technology  substitution 
or  vendor  replacement,  thereby  reducing  competition  due  to  vendor  lock-in.  This  works 
against  defense  affordability.  In  contrast,  emerging  online  repositories  offer  different  kinds  of 
software  components  with  different  functional  capabilities  (described  earlier)  along  with 
different  IP  licenses  and  end-user  licenses  (e.g.,  low-cost,  per-user  licenses).  These 
repositories  of  software  components  represent  a  means  for  increased  competition  and 
affordability  but  are  subject  to  different  acquisition,  development,  or  integration  processes 
that  are  just  coming  to  light.  Accordingly,  we  believe  that  streamlining  the  acquisition 
process  for  secure,  component-based  OA  systems  requires  that  IP  license  cost  obligations 
(e.g.,  license  fees  for  end-user  agreements)  and  license  management  regimes  need  to  be 
incorporated  into  process  measurement  and  assessment,  process  redesign  and  evolution, 
and  the  design  of  new  acquisition  processes.  This  is  also  a  subject  for  further  acquisition 
research — but  one  offering  practical  near-term  consequences. 

Conclusions 

In  this  paper,  we  presented  our  current  results  from  an  ongoing  investigation  of  how 
best  to  acquire  secure  OA  software  systems.  These  systems  incorporate  SPL  practices  that 
include  closed  source  proprietary  software  and  OSS  components,  where  such  components 
and  overall  system  configurations  are  subject  to  different  security  requirements.  The 
combination  of  SPLs  and  OSS  components  within  secure  OA  systems  represents  a 
significant  opportunity  for  reducing  the  acquisition  costs  of  software-intensive  systems  by 
the  DoD  and  other  government  agencies.  Through  our  research  efforts,  we  seek  to  make 
the  acquisition  of  secure,  component-based  OA  systems  a  simpler,  more  transparent,  and 
more  tractable  process.  Such  a  process  must  be  easy  to  explicitly  model,  share,  reuse, 
adapt,  and  streamline  for  different  system  application  domains.  Our  goal  was  to  identify 
ways  and  means  for  how  to  realize  cost  reductions  and  improve  acquisition  workforce 
capabilities  in  ways  that  address  BBP  initiatives  associated  with  the  move  to  OA  systems 
and  licenses  (DAU,  2012). 

In  this  paper,  we  identified  different  ways  and  means  for  how  to  streamline  the 
acquisition  process  for  secure  OA  software  systems  through  a  focus  on  doing  more  with 
limited  resources.  Central  to  our  approach  was  our  effort  to  identify  and  characterize  new 
ways  and  means  for  acquisition  process  measurement  and  assessment,  process  redesign 
and  evolution,  the  design  of  new  acquisition  processes,  and  the  incorporation  of  cost  factors 
and  cost  drivers  as  an  element  in  new  acquisition  processes.  Along  the  way,  we  paid 
particular  attention  to  revealing  how  licensing  practices  for  emerging  online  software 
component  marketplaces  can  affect  cost  in  ways  that  either  hamper  or  better  the  buying 
power  of  acquisition  programs.  Consequently,  we  sought  to  identify  possible  next  steps  for 
new  acquisition  research  that  can  further  accelerate  efforts  to  improve  competition  and 
defense  affordability  as  well  as  empower  the  acquisition  workforce  going  forward,  in  ways 
aligned  with  BBP  initiatives. 
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Abstract 

The  lack  of  focus  on  complexity  issues  in  System  of  Systems-related  acquisitions  prevents 
effective  support  for  Better  Buying  Power  (BBP)  targets  of  affordability,  innovation,  increased 
productivity,  and  healthy  competition  in  reducing  costs  and  improving  delivery  of  promised 
performance.  The  impetus  is  to  provide  the  necessary  analytical  frameworks  and  associated 
tools  that  enable  better  informed  decisions  in  support  of  BBP  objectives.  This  paper  extends 
our  previous  work  in  robust  portfolio  optimization  and  adopts  a  multi-period  portfolio 
management  approach  to  support  the  objectives  of  BBP.  Derived  from  the  financial 
engineering  and  operations  research  literature,  robust  multi-period  portfolio  management 
principles  provide  a  decision-making  framework  that  balances  performance  of  a  “portfolio”  of 
systems,  constituting,  for  example,  a  system  of  systems,  against  potential  risks.  The 
framework  also  balances  short  versus  long  term  gains  through  its  multi-period  formulation.  An 
illustrative  example,  using  a  Littoral  Combat  Ship-inspired  naval  warfare  scenario, 
demonstrates  application  of  the  approach  and  potential  use  for  acquisition  practitioners. 

Introduction 

The  U.S.  Department  of  Defense  (DoD)  has  emphasized  a  need  for  Better  Buying 
Power  (BBP)  initiatives  in  tackling  issues  of  increasing  costs,  schedule  growth,  and  reduced 
productivity.  The  success  of  BBP  policies  in  reducing  costs  have  been  well  documented  for 
a  variety  of  cases  that  include  the  acquisition  of  Navy  destroyers,  reduction  in  production 
rates  for  the  E-2D  Hawkeye  program,  and  cutting  cycle  times  and  cost  of  ammunitions 
through  an  improved  small  business  acquisition  strategy.  However,  the  complexities  of 
modern  platforms  that  interact  as  a  system  of  systems  (SoS;  Maier,  1998)  present  the  risk  of 
cascading  modes  of  failure;  this  is  due  to  the  highly  interdependent,  yet  operationally  and 
managerially  independent,  interactions  between  the  constituent  systems.  The  desire  to 
promote  adequate  competitions  and  growth  of  technological  options  in  developing  military 
capabilities  has  further  increased  the  complexity  of  the  acquisition  process.  This  increase  in 
complexity  now  includes  the  need  to  account  for  competitive  elements  in  contracting, 
improving  productivity,  and  reducing  unnecessary  redundancies.  The  management  of  Major 
Defense  Acquisition  Programs  (MDAPs)  through  a  “should  cost-will  cost”  imperative 
becomes  increasingly  difficult  as  acquisition  decisions  must  carefully  balance  performance 
and  risk,  and  time. 

The  acquisition  of  systems  with  an  SoS  capability  in  mind  increases  the  complexity. 
Current  tools  especially  for  this  problem  context  are  lacking.  Figure  1  shows  an  abstraction 
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of  the  hierarchical  and  complex  relationships  among  the  individual  layers  of  systems  in 
satisfying  requirements  and  consequently,  desired  overarching  SoS  level  capabilities. 


Figure  1.  System  of  Systems  Hierarchy 


The  DoD  (2012)  Defense  Acquisition  Guidebook  (DAG)  and  DoD  System  of  Systems 
SE  guide  provides  fundamental  guidance  in  tackling  SoS-related  acquisitions;  however, 
these  greatly  lack  the  necessary  depth  and  decision  tools  in  support  of  BBP  objectives.  The 
lack  of  an  effective  decision  support  framework  for  managing  acquisition  risks  has  led  to 
cascading  cost  overruns,  schedule  delays,  and  even  program  cancellations.  Examples  of 
these  effects  have  been  observed  in  several  programs  such  as  the  Joint  Strike  Fighter,  U.S. 
Army  Future  Combat  Systems  (FCS;  Gilmore,  2006),  and  U.S.  Navy  Littoral  Combat  Ship 
(LCS;  O’Rourke,  2011)  programs.  Computational  decision  support  frameworks  are  needed 
to  adequately  deal  with  the  complexity  of  interconnected  acquisition  domains  and  to  identify 
optimal  collections  of  systems  that  mitigate  cascading  risks. 


Investment  Portfolio  Management:  A  Path  to  Better  Buying  Power 

Portfolio  management  techniques  have  been  successfully  applied  to  the 
management  of  strategic  “portfolios  of  systems”  in  military  acquisitions;  this  includes 
application  of  Real  Options  (RO)  theory  and  metrics  such  as  Knowledge-Value  Added  (KVA) 
that  account  for  the  value  added  by  human  and  IT  investments  (Komorovski,  Housel,  Horn, 

&  Mun,  2006).  Work  by  Mun  (2005)  has  developed  an  eight-phase  process  to  addressing 
portfolio  management  of  strategic  assets.  Work  by  Giachetti  (2012)  has  applied  stochastic 
techniques  to  managing  military  investments.  Previous  research  funded  by  the  Naval 
Postgraduate  School  (NPS)  and  presented  at  the  2012  NPS  Acquisition  Research 
Symposium  (Davendralingam,  Mane,  &  DeLaurentis,  2012),  has  focused  on  a  robust 
portfolio  management  problem  of  maximizing  a  warfighter  SoS  portfolio  performance  index 
while  preserving  budgetary  and  compatibility  constraints  of  underlying  military  assets.  The 
robust  portfolio  work  complements  prior  research  efforts  to  include  algorithmic  advances, 
explicit  consideration  of  data  uncertainty,  and  inclusion  of  SoS  architectural  information 
within  a  robust  investment  portfolio  framework.  The  robust  portfolio  methodology  is  adapted 
from  financial  engineering  literature  and  leverages  potential  gains  in  overall  SoS  capability 
against  cost  and  developmental  risks  in  selecting  “baskets”  of  compatible,  interdependent 
systems. 

Risks  and  capabilities  associated  with  system  interdependencies  can  span  the 
functional  or  physical  spaces  of  the  SoS  construct  and  is  subject  to  uncertainty.  The 
developed  strategy  supports  acquisitions,  both  in  the  pre-  and  post-milestone  B  phases,  and 
considers  current  initiatives  such  as  open  architecture  (OA)  and  competitive  contracting 
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(e.g.,  fixed-price  initiatives)  in  improving  affordability  and  BBP  objectives  while  considering 
evolving  military  requirements.  Work  in  this  research  extends  the  robust  portfolio  approach 
to  include  a  multi-period  portfolio  perspective.  The  multi-period  portfolio  optimization 
approach  draws  upon  a  rich  history  of  algorithmic  development,  as  noted  in  operations 
research-related  literature  (Powell,  2011;  Bertsimas  &  Pachamanova,  2008;  Bertsekas, 
2005;  Fabozzi,  Kolm,  Pachamanova,  &  Focardi,  2007;  Tutuncu  &  Cornuejols,  2007).  Its 
roots  stem  from  sequential  decision-making  areas  known  broadly  as  dynamic  programming 
or  stochastic  programming  and  adapts  control  theory  methodologies  to  the  dynamic 
management  of  resources  in  the  interest  of  maximizing  (or  minimizing)  some  given  metric. 
Stochastic  programming  focuses  on  issues  of  uncertainty  whereas  dynamic  programming 
relates  to  the  optimality  of  making  sequential  decisions;  however,  there  has  been  a  large 
degree  of  overlap  and  exchange  between  the  two  areas.  Algorithmic  development  in  these 
areas  has  been  applied  to  a  range  of  real-world  dynamic  decision-making  problems  that 
range  from  financial  portfolio  management  to  real-time  control  of  vehicles. 

A  Multi-Period  Decision-Making  Framework 

The  multi-period  portfolio  approach  enhances  the  robust  portfolio  decision-support 
framework  and  better  enables  optimal  acquisitions  of  systems  in  maximizing  SoS-wide 
capabilities.  The  construction  of  an  appropriate  dynamic  policy,  in  the  context  of  an 
acquisition  management  problem,  translates  to  identifying  actions  that  balance  the  potential 
gains  in  SoS  capabilities  against  developmental  risks  (e.g.,  cost  and  schedule  growth  risks) 
over  a  specified  time  horizon.  Figure  2  is  an  abstraction  of  the  evolution  of  a  “portfolio  of 
systems”  that  constitute  an  SoS,  as  part  of  the  wave  model  (Dahmann,  Rebovich,  Lane, 
Lowry,  &  Baldwin,  2011). 


Figure  2.  Wave  Model  Relation  to  Portfolio  Evolution 


The  wave  model  is  an  extension  of  the  Department  of  Defense  guidelines  on 
systems  engineering  (SE)  for  an  SoS  that  translates  SoS  SE  core  elements, 
interrelationships,  and  decision-making  artifacts  from  a  previous  “Trapeze”  model  to  a  time- 
sequenced  model  representation  (Dahmann  et  al. ,  2011).  These  architectural  decisions 
involve  the  acquisition  of  assets  in  evolving  the  SoS  capabilities  to  meet  core  objectives;  the 
SoSE  architect’s  role  is  to  explore  the  trade  space  across  multiple  operationally  independent 
domains  in  determining  suggested  architectural  modifications  (add/remove  assets)  in 
evolving  the  SoS. 
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A  Multi-Period  Decision  Framework 

The  objective  of  the  robust  multi-period  portfolio  framework  is  to  allow  for 
mathematical  rigor  of  algorithmic  techniques,  transparent  to  the  end  user/practitioner,  to 
support  SoS-level  acquisition  decisions  through  identification  of  optimal  “portfolios”  of 
systems  to  be  acquired  in  pursuit  of  desired  SoS  capabilities.  While  the  acquisition  process 
spans  operationally  and  managerially  independent  defense  groups,  the  tools  and 
frameworks  envisioned  to  support  these  aspects  are  aimed  at  providing  adequate  trade 
space  exploration  capabilities.  These  explorations  require  a  domain  agnostic  framework, 
and  hence  intuitively  resonate  with  the  idea  of  treating  the  collection  of  systems  across 
domains  as  a  “portfolio”  of  systems  in  the  SoS. 

This  is  often  the  case  in  operations  research  and  financial  engineering  applications, 
where  underlying  mathematical  optimization  frameworks  are  used  to  drive  decision  support 
software  in  assisting  decision-makers  (e.g.,  policymakers,  investment  specialists)  in 
performing  acquisition  analysis.  The  concept  naval  warfare  scenario  in  this  paper 
demonstrates  the  application  of  the  multi-period  portfolio  framework  in  managing  the 
sequential  acquisitions  needed  to  propagate  required  capabilities  while  minimizing 
operational  and  developmental  risks.  The  method  illustrates  the  identification  of  optimal 
evolution  of  interconnected  systems  that  cohesively  function  in  providing  an  overarching 
SoS-wide  capability.  A  robust  optimization  approach  to  the  multi-period  portfolio  formulation 
addresses  issues  of  data  uncertainty. 

Development  of  a  Multi-Period  Investment  Portfolio  Model 

The  acquisition  (and  removal)  of  systems  in  an  evolving  an  SoS  inherently  involves  a 
timeline  of  sequentially  executed  decisions.  Decisions  made  at  each  epoch  affect  the 
decision  options  of  future  states,  thus  affecting  long  term  performance  and  risks  of  the  SoS 
gamut.  The  translation  of  these  sequential  decisions  to  the  context  of  a  multi-period 
investment  model  requires  an  adequate  description  of  node  (system)  attributes;  this  ensures 
the  selection  of  feasible  portfolios  that  satisfy  nodal  requirements  and  minimize  cascading 
risks.  Figure  3  shows  modeled  generic  behaviors  for  systems  being  considered  in  an  SoS 
portfolio. 

System-of-Systems  Modeling 


In  Figure  3,  the  capabilities  of  an  existing  SoS  (initial  blue  nodes)  have  the  potential 
to  evolve,  based  on  potential  connections  to  yet-to-be  acquired  systems  (dashed  lines  and 
nodes).  At  each  decision  epoch,  the  practitioner  utilizes  a  decision-making  framework  (such 
as  the  multi-period  portfolio  framework)  to  evaluate  the  value  and  risks  involved  in  the 
potential  acquisitions  of  new  systems  (denoted  by  red  dashed  lines).  The  resulting  new 
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collection  of  systems  that  comprise  the  new  SoS  construct  now  includes  the  addition  of  the 
new  systems. 

An  SoS  is  treated  as  a  set  of  generic  discrete  nodes  with  the  following  attributes: 

•  Capability  (Outputs):  Nodes  have  finite  supply  of  capabilities  that  are  limited 
by  quantity  (e.g.,  total  power  output  of  generator  systems). 

•  Requirements  (Inputs):  Nodes  have  individual  requirements.  Requirements 
are  fulfilled  by  receiving  capabilities  from  other  nodes  that  can  fulfill  said  set 
of  requirements  (e.g.,  a  high  powered  AMDS  radar  requirement  of  energy  can 
be  fulfilled  by  multiple  generators). 

•  Compatibility:  Nodes  can  only  connect  to  other  nodes  based  on  a  pre- 
established  set  of  rules  (e.g.,  AMDS  radar  can  only  accept  power  from  high 
capacity  nuclear  reactor  systems  on  specific  ships). 

Multi-Period  Investment  Portfolio  Formulation 

The  problem  statement  for  a  multi-period  investment  portfolio  is  translated  to  the 
language  of  mathematical  programming.  The  process  begins  with  the  definition  of  two  main 
elements  of  a  mathematical  program,  namely,  the  objective  function  and  constraints.  The 
objective  function  is  a  mathematical  expression  that  is  formulated  to  reflect  a  key 
performance  metric  of  the  system  to  be  maximized  (or  minimized).  Typical  formulations  of 
the  objective  function  seek,  for  example,  to  minimize  direct  costs  of  operating  a  fleet  of 
aircraft.  For  an  SoS,  the  objective  function  reflects  a  chosen  measure  of  performance  and 
associated  costs.  The  second  important  aspect  of  a  mathematical  program  is  the  formulation 
of  the  constraints.  The  constraints  reflect  physical,  resource,  and  behavioral  aspects  of  the 
systems  as  mathematical  expressions.  Our  initial  framework  for  a  multi-period  portfolio 
considers  a  long  term  horizon  of  acquisitions  with  discrete  decision  steps  that  denote 
periods  of  “investment”;  these  investments  involve  the  addition/removal  of  individual 
systems  that  comprise  the  overall  SoS  network. 

The  following  mathematical  program  describes  a  preliminary  framework  for  the  multi¬ 
period  acquisition  problem: 


subject  to: 


(1) 


yB  _  yB  I  jjB  I  i tB 
Aq,t  ~  Aq,t- 1  +  uq,t  +  vq,t 

ptrans  _  pBijB  <  pS  t rS 

Lt  ~  Lq  uq,t  Lq  v q,t 


(2) 

(3) 


T 


Zftrans 
Lt 

t= 0 


<  Budget 


(4) 


^  SqtC  Xq,t 


> 


(Satisfy  Requirements  at  each!)  (5) 

<7 
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(X?t +. .  +X%,t)  .  =  MJ  t  j  —  1 ...  k  (Package  System  Compatibility)  (6) 
Uq,t’Vq,t  e  [0,1]  t=0. . .T  (time  steps)  (7) 


where: 

w  -  weighting  factor  vector  that  weights  the  importance  of  constituent  capabilities  of 
index 

Rc  -  baseline  capability  level  for  each  of  the  capabilities  that  contribute  to  index 

Cq  t  -  cost  of  acquiring  system  ( q )  at  time  (f) 

C?  -cost  of  retiring  system  ( q )  at  time  (f) 

Equation  1  is  the  weighted  objective  function  that  seeks  to  maximize  the  end 
developed  SoS  performance  index.  Here,  the  index  is  related  to  the  final  state  of  the 
portfolio  ( t  =  T)  and  is  weighted  according  to  the  value  that  each  capability  (C)  contributes  to 
the  index  (however,  this  can  naturally  reflect  maximization  of  each  stage,  if  necessary).  The 
index  is  normalized  by  referencing  it  to  some  chosen  reference  capability  set  ( Rc ).  Equation 

2  reflects  the  evolutionary  nature  of  the  portfolio  of  chosen  systems  ( q )  at  time  ( t ), 
represented  by  the  decision  vector  Here,  the  decision  vector  is  binary,  to  reflect  discrete 
system  choices;  however,  a  more  general  setting  can  allow  for  the  variables  to  be 
continuous  in  nature. 

The  terms  Uq  t,  and  V®t  reflect  decisions  to  “acquire”  and  “remove/retire”  individual 
systems  respectively,  in  the  portfolio  of  systems  at  each  decision  epoch  of  time  ( t ).  Equation 

3  captures  the  “transactional”  costs  at  each  stage;  this  means  that  decisions  to 
acquire/remove  systems  translate  to  costs  associated  with  each  that  are  accrued  at  each 
time  step.  In  acquisitions,  the  removal  cost  translates  to  a  salvage/swap  cost  for  changing 
out  individual  systems  whereas  the  “acquire”  cost  is  simply  the  cost  of  purchasing  and 
integrating  a  new  system.  Equation  4  ensures  budgetary  balance  for  total  costs 
(transactional  and  acquisition)  that  occur. 

Equation  5  ensures  that  the  total  “capabilities”  from  systems  acquired  satisfy  the 
requirements  that  individual  systems  may  have;  for  example,  there  must  be  adequate  power 
generating  systems  selected  to  support  selected  communications  systems  that  provide 
some  system-wide  communications  capability.  Conditions  for  Equation  5  can  be  enforced  at 
each  time  step  (t)  or  at  the  final  stage  (t  =  T),  depending  on  requirements  at  each  time  step. 
Equation  6  enforces  compatibility  constraints  as  binary  conditions  for  a  total  of  ( k )  set  of 
rules;  for  example,  the  constraint  that  only  one  engine  can  be  selected  to  generate  power 
would  translate  to  a  constraint  of  xt  +  x2  =  1  where  (xl,x2)  are  binary  variables.  The  rules 
can  be  applied  across  decision  epochs,  reflecting  the  need  to  have  prior  systems  in 
existence,  before  particular  upgrades  can  be  implemented  in  future  time  steps.  Equation  7 
states  that  the  decision  variables  are  binary  and  that  the  time  window  consists  of  discrete 
steps  from  t  =  0  to  a  final  time  t  =  T.  The  problem  formulation  of  Equations  1-7  constitutes 
a  binary  integer  program,  for  which  efficient  methods  of  solution  and  commercial  solvers  are 
available. 

Robust  Multi-Period  Investment  Portfolio 

The  multi-period  formulation  of  Equations  1-7  are  deterministic  and  do  not  consider 
uncertainties  in  the  data.  Real  world  systems  are  inherently  driven  by  uncertainty  and  thus 
challenge  the  optimality  (and  feasibility)  of  decisions  made  under  deterministic  assumptions. 
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Research  in  mathematical  programming  has  progressively  focused  more  on  the 
development  of  robust  optimization  methods  to  deal  with  manifestations  of  uncertainty. 
Robust  optimization  seeks  to  find  solutions,  to  uncertain  mathematical  programming 
problems,  that  are  less  sensitive  to  parametric  variations  in  the  problem  being  solved.  We 
consider  uncertainties  in  the  data  for  Equations  1-7,  namely  in  the  “transaction  costs”  of 
Equations  3  and  4  that  reflect  system  addition  and  removal  costs.  We  also  consider 
uncertainties  in  the  capabilities  of  each  system  available. 

The  consideration  of  the  uncertainty  in  the  multi-period  formulation  requires  the  use 
of  robust  optimization  methods  for  solution.  There  are  a  range  of  methods  that  can  address 
the  uncertain  linear  structure  of  the  resulting  optimization  problem;  however,  we  adopt  the 
Bertsimas-Sim  (correlated  case)  approach  for  our  preliminary  multi-period  framework.  The 
Bertsimas-Sim  method  (Bertsimas  &  Sim,  2004)  is  a  robust  optimization  approach  to  solving 
linear  optimization  problems  with  uncertain  data.  The  method  allows  for  a  flexible 
adjustment  in  the  level  of  conservatism  of  the  robust  solutions  (termed  the  Price  of 
Robustness)  in  terms  of  probabilistic  bounds  of  constraint  violations. 

We  consider  the  following  to  be  a  general  uncertain  linear  program: 


subject  to: 


maximize  cTx 
Ax  —  b 


(8) 

(9) 


x  >  0  (10) 

Where  values  aij  of  matrix  A  are  uncertain  and  exist  in  the  nominally  symmetric 
bounds  of  [ a y  -  a^.a^  +  ai;-].  The  uncertainties  are  treated  as  constraint-wise  uncertainties. 
In  the  correlated  case,  the  uncertainties  are  modelled  as  the  following  equation: 

oij  =  an  +  ^  vik  gkj  (ii) 

ke  Ki 

where  rjik are  the  independent  and  symmetric  random  variables  [-1,  1],  and  there  are  k 
number  of  uncertain  sources.  The  robust  optimization  problem  to  the  correlated  case  can  be 
written  as  the  following  linear  optimization  problem  (Bertsimas  &  Sim,  2004): 

maximize  cTXj  (12) 


subject  to: 

^ AijXj  H-z^  +  ^  pij  <  bi 

j  je  h 

zi+pij>yj 

-yj<  ^  grp]  ^  yj 

ieJi 

Ij  <  Xj  <  yj 


(13) 

(14) 

(15) 

(16) 
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(17) 


Vij.yij-Zij  ^  o 

where  p  ,y  ,zy  are  the  dual  variables  associated  with  the  dual  problem  of  the  nonlinear 
formulation  of  the  Bertsimas-Sim  method  (See  Bertsimas  and  Sim  [2004]  for  full  derivation), 
and  J  is  the  set  of  uncertain  coefficients.  The  conservatism  term,  I),  is  adjusted  to  control 
probabilistic  guarantees  of  constraint  (i)  violation.  For  example,  changing  T,  for  linear 
constraints  that  dictates  power  distribution  flow  over  a  network,  controls  the  probability  of  net 
power  being  supplied  at  a  prescribed  level  of  cost.  The  constraint  violation  probability 
bounds  for  individual  constraints  can  be  approximated  using  the  following  De  Moivre 
approximation  of  the  binomial  distribution  (Bertsimas  &  Sim,  2004): 

(18) 

where  n  is  the  |y._J  and  O  is  the  normal  cumulative  distribution  function.  The  manipulation  of 
T  in  controlling  the  probability  of  constraint  violation,  allows  for  an  intuitive  interpretation  of 
the  conservatism  of  solutions  generated  and  permits  practitioners  the  means  of  assessing 
solution  performances  against  associated  risk  in  terms  of  individual  constraint  violations. 

Robustification:  Bertsimas-Sim  (Correlated)  Approach 

The  robust  (correlated)  implementation  of  the  Bertsimas-Sim  approach  in  Equations 
1 1-17  is  applied  to  the  multi-period  model  of  Equations  1-7.  The  following  equations 
described  the  robustified  budget  constraints  for  the  multi-period  model,  in  particular  the 
context  of  budget  feasibility,  expressed  earlier  in  Equation  4: 


T 


Xq,t=r  +  CcKt  +  zr  +  ^  P;  <  Budget 

(19) 

t=o  .  jeJi 

'CTXj ' 

+ 

IV 

(20) 

-yj<  y  gup]  ^  yj 

(21) 

jeJi 

lj  <  Xj  <  yj 

(22) 

Pij.yij.Zij  ^  0 

(23) 

where  xj  is  the  concatenated  decision  vector  {X%it=T  ^=0,1,2}  associated  with  all 
transactions  (t  =  0,1,2). 

Interpretation  of  Risk 

The  inclusion  of  correlation  information  reflects  an  important  contribution  where 
protection  levels  of  each  robust  constraint,  in  the  non-correlated  case  assumes  the 
simultaneous  worst-case  scenarios  at  the  uncertainty  bounds-a  condition  that  is  highly 
improbable.  The  correlated  case  accounts  for  the  simultaneous  “movements”  in 
performance  and  risks  across  the  capabilities  of  individual  assets.  Prior  research  has  utilized 
a  mixed  integer  semidefinite  programming  (MISDP)  approach  to  dealing  with  uncertainties  in 
the  covariance  matrix,  a  matrix  that  is  associated  with  variances  (risk)  in  system 
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development  time.  However,  there  are  very  limited  solvers  that  are  able  to  solve  MISDPs, 
which  limits  practical  implementation,  despite  some  of  the  computational  advantages  in 
dealing  with  uncertainty. 

Concept  Application:  Naval  Acquisition  Scenario 

The  Naval  Acquisition  Scenario  is  based  on  the  Littoral  Combat  Ship  (LCS)  model 
(LCS,  2011).  The  LCS  (Figures  4  and  5)  is  a  naval  combat  vessel,  developed  by  Lockheed 
Martin  and  General  Dynamics,  as  a  result  of  the  Navy’s  dual  contracting  efforts  to  reduce 
cost  through  competition.  The  design  of  these  ships  seeks  to  provide  a  more  agile  and  cost- 
effective  solution  to  various  near  shore  environment  missions.  These  missions  are  executed 
through  use  of  interchangeable  ship  packages  that  include  Mine  Warfare  (MIW),  Anti- 
Submarine  Warfare  (ASW),  and  Surface  Warfare  (SUW).  The  highly  modular  design  of  the 
platform  allows  for  a  great  degree  of  operational  flexibility.  The  modularity  also  translates  to 
the  ability  for  open  architecture  and  small  business  initiatives  to  be  brought  to  bear  in 
reducing  program  costs  and  improving  competition.  Our  ongoing  work  in  this  paper  assumes 
an  LCS-inspired  scenario  as  representative  “simple”  SoS  model  where  the  objective  is  to 
identify  potential  sequence  of  investment  decisions  and  the  corresponding  end  collection  of 
systems  that  can  best  maximize  core  capabilities  of  the  SoS  mission  (in  this  case,  MIW, 
ASW,  and  SUW). 


Our  highly  simplified  model  consists  of  a  hypothetical  list  of  candidate  systems,  listed 
in  Table  1 ,  that  are  available  to  the  Navy  for  acquisition.  Although  the  numbers  presented  in 
the  table  are  fictitious,  the  salient  features  of  capability,  requirements,  cost,  and  uncertainty 
are  nevertheless  represented.  Each  subset  of  systems  (listed  by  categories  of  ASW,  MCM, 
SUW,  Seaframe,  Comm)  represents  a  subset  collection  of  systems  that  are  available  in 
meeting  the  needs  of  each  category.  The  ASW,  MCM,  and  SUW  categories  are  the  core 
LCS  mission  packages.  “Seaframe”  reflects  the  ship  seaframe  support  options,  and 
“Communications”  represents  the  support  communications  systems  available  for 
deployment.  The  first  five  columns  show  capabilities  of  each  system,  and  their  respective 
numerical  valuations.  Columns  6  and  7  are  the  Power  and  Communications  requirements 
needed  for  operation  of  the  listed  systems,  in  providing  the  capabilities  listed.  Also  listed  are 
the  acquisition  (buy)  and  retiring  (sell/salvage)  costs,  along  with  the  estimated  uncertainty  of 
each  cost.  We  consider  uncertainty  in  costs  for  this  simplified  problem;  however,  more 
general  uncertainty  in  capabilities  or  requirements  can  be  introduced  in  the  same  fashion. 


Figure  4.  Concept  of  Operations 

Note.  Image  taken  from  presentation  slides  by  RDML  Vic  Guillory  of  OPNAV  at  the  Mine  Warfare 
Association  Conference  (titled  “Littoral  Combat  Ship”),  May  8,  2007. 
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Figure  5.  General  Dynamics  Independence  Class  LCS 


Table  1.  LCS  Candidate  System  Scenario 


Category 

System 

Weapon 

Strike 

Range 

Surface  Anb  Mine 

Detect  on  Detect!  on 

Range  Range 

Comm  Power  Power  Comm 

Bandwith  Bandwith  Required  Required 

Acquisition 

Cost 

Retiring 

Cost 

uncertainty  uncertainty 
Acquisition  Retiring 
Cost  Cost 

ASW 

Van  Ale  Depth 

0 

50 

0 

0 

0 

95 

100 

L006-+05 

1005405 

9.54E401 

3.045401 

Multi  Fen  Tow 

0 

40 

0 

0 

0 

90 

120 
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Naval  Acquisition  Scenario:  Results 

The  problem  statement  for  the  above  LCS-inspired  acquisition  problem  is  formulated 
as  a  mathematical  program  that  follows  the  robustified  formulation  of  Equations  1-7.  The 
resulting  problem  is  then  solved  for  varying  values  of  conservatism,  T i5  to  reflect  a  range  of 
dynamically  evolving  acquisitions,  at  each  prescribed  level  of  conservatism.  Here,  we 
assume  conservatism  in  dealing  with  the  costs  uncertainties  of  acquisitions;  each  chosen 
value  of  T  (here,  three  values)  in  this  context  thus  reflects  the  probability  of  budget  overruns 
occurring  due  to  the  associated  costs  uncertainties  in  each  stage  of  acquisition.  We  assume 
a  three-stage  (t=  0,1,2)  acquisition  process,  where  the  systems  listed  in  Table  1  can  be 
acquired  or  retired  at  each  stage,  culminating  to  a  final  “portfolio”  of  assets  at  the  end  of 
stage  3  (f= 2).  Acquisition  or  retirement  of  these  systems  is  subject  to  a  prescribed  set  of 
rules  that  govern  their  compatibility  and  availabilities  in  time  (systems  only  available  at 
specific  epochs)  as  reflected  in  Equation  6  of  the  problem  formulation.  Figure  6  shows  the 
SoS  performance  frontier  tradeoff  against  degree  of  conservatism  in  the  budget  constraint. 
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Figure  6  highlights  three  dynamic  portfolios  at  conservatism  level  of  T  =  0.001 , 0.5, 
and  1  respectively;  increasing  values  of  T  indicate  a  higher  degree  of  conservatism.  Each 
point  corresponding  to  a  particular  chosen  level  of  conservatism  reflects  a  sequence  of 
acquisition  decisions  that  lead  to  the  final  portfolio  performance  index  denoted  on  the  graph. 
The  sequence  of  acquisitions  for  each  level  of  conservatism  is  shown  in  Table  2,  where  “1” 
denotes  a  decision  to  acquire  a  particular  system  at  that  time  step,  t.  Figure  7  shows  the 
normalized  capability  index  for  each  subset  of  capabilities  that  comprise  the  index  (in  this 
case,  weapons  strike  range,  surface  detection  range,  and  anti-mine  detection  range)  of  each 
of  the  optimal  points  in  Figure  6. 

The  results  in  Table  2  indicate  evolving  portfolio  of  systems  where  individual  systems 
are  acquired  and  retired  throughout  the  decision  epochs,  preserving  the  satisfaction  of 
requirements,  towards  maximizing  the  end  goal  of  the  overall  SoS  portfolio  at  time  t  =  T. 
Retirements  are  denoted  by  the  evolution  from  a  previously  selected  state  (e.g.,  xjt= 1  at 
t  =  2)  to  a  state  of  (e.g.,  xjt  =  0  at  t  =  3).  At  a  high  level  of  conservatism  (r  =  1.0),  we 
observe  the  expected  case  of  the  portfolio  being  constant,  where  the  initial  investments  are 
held  over  the  entire  decision  horizon  without  any  retirement  or  further  acquisitions;  this 
reflects  the  condition  where  risks  associated  with  the  buy/retire  transactions  are  deemed  to 
be  too  great,  hence  prompting  the  selection  of  a  lower  capability  but  less  financially  risky 
acquisition  strategy.  At  the  low  and  mid-levels  of  conservatism,  there  is  a  possibility  of 
sequential  acquisitions,  subject  to  the  availability  and  compatibility  rules  between  systems, 
that  can  result  in  higher  performing  portfolios  but  at  higher  prescribed  level  of  acquisition 
risk. 


The  results  of  Table  2  and  Figure  7  afford  practitioners  a  candid  view  of  the 
“topology”  of  acquisitions  that  can  optimally  be  made  over  time,  assuming  a  tolerance  of  risk 
for,  in  this  case,  and  budgetary  risk.  The  risk  uses  correlated  information  on  the  costs  and  is 
quantified  as  the  probability  associated  with  the  budget  constraint  violation.  The  analysis 
result  presented  can  be  useful  to  decision-makers  in  assessing  the  potential  dynamic 
purchasing/retirement  decisions  that  need  to  be  made  in  view  of  quantifiable  uncertainties.  It 
also  allows  the  decision-maker  to  assess  the  trade-offs  between  performance  and  risks  in 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  635- 


decisions  at  each  epoch  of  the  acquisition  process,  while  bearing  independencies  and 
system  compatibilities  in  mind.  The  mapping  of  the  dynamic  acquisition  trade-space  can 
also  better  inform  independent  acquisition  groups,  within  an  SoS,  on  the  potential  actions 
that  various  collaborative  acquisition  strategies  can  have  on  the  overall  scheme  of 
development. 


Table  2.  Portfolio  Evolution  at  Varying  Conservatism 
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Figure  7.  SoS  Capability  Spread  at  Varying  Conservatism 
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Conclusions  and  Future  Work 

The  development  of  a  portfolio  of  systems  to  serve  in  an  SoS  context  is  a  difficult 
endeavor.  Complex  interdependencies  and  uncertainties  abound  in  both  capabilities  and 
requirements  of  its  constituent  systems.  There  is  an  absence  of  adequate  frameworks  and 
tools  to  enable  effective  navigation  of  the  resulting  trades-spaces.  Research  in  this  paper 
presents  a  preliminary  framework  for  a  robust  multi-period  portfolio  approach  to  facilitate 
selection  of  systems  for  acquisition  in  this  context.  The  framework  is  naturally  based  on 
multi-period  portfolio  and  robust  optimization  techniques,  and  it  has  shown  promise  in 
assessing  the  impact  that  various  degrees  of  risk  aversion  (here,  conservatism)  on 
acquisition  related  decision  epochs. 

The  simple  LCS-inspired  Naval  Warfare  Scenario  demonstrates  application  of  the 
framework;  the  objective  is  to  identify  optimal  acquisition  decisions  (buy/retire)  at  each 
decision  epoch,  assuming  various  levels  of  conservatism  in  budget  violations.  The  analysis 
affords  practitioners  a  candid  view  of  the  dynamic  acquisition  trade-space  and  allows  for  the 
selection  of  systems  at  the  prescribed  levels  of  accepted  conservatism.  In  the  larger  context 
of  acquisition  affordability  objectives,  the  algorithmic  framework  established  here  has  direct 
bearing  on  BBP  focus  areas,  as  listed  in  Table  3. 


Table  3.  BBP  Contributions 


Better  Buying  Power 

Focus  Area 

Potential  Contribution  of  Multi-Period  Portfolio  Approach 

Achieve  Affordable  Programs 

•  Robust  decision-making  in  a  multi-period  setting 
enables  mitigation  of  risks  and  planning  of 
development  steps 

Control  Lifecycle  Costs 

•  Robust  multi-period  portfolio  accounts  for 

uncertainties  in  transactional  costs  at  each  stage  of 
the  decision  horizon. 

Incentivize  Productivity  and 
Innovation  &  Promote  Effective 
Competition 

•  Metrics  such  as  KVA  and  piece-wise  linear  modeling 
of  incentivizations  in  a  multi-period  setting  can 
provide  robust  management  of  investments  for  non¬ 
tangible  investments  and  incentivizations 

•  Enables  effective  management  of  larger  set  of 
acquisition  possibilities  (e.g.,  contributions  from 

SBIRs,  open  architectures) 

Our  future  work  will  leverage  the  robust  multi-period  approach  by  incorporating 
relevant  metrics  and  perspectives,  as  listed  in  Table  3  above,  to  more  explicitly  account  for 
sequential  decision-making  processes  in  promoting  affordability  in  defense  acquisitions. 
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Abstract 

Acquisition  governance  currently  confronts  two  problems:  the  growing  size  and  complexity  of 
systems-of-systems  capabilities  and  the  limited  effectiveness  of  existing  governance  models 
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to  ensure  the  on-cost  and  on-schedule  delivery  of  those  capabilities.  The  Center  for  Strategic 
and  International  Studies  (CSIS)  is  engaging  in  research  on  systems-of-systems  acquisition 
governance  best  practices  that  could  help  the  defense  acquisition  community  overcome 
some  of  these  problems.  This  report  provides  an  update  on  the  progress  of  that  effort.  It 
reviews  the  evolution  of  acquisition  governance  models  throughout  the  history  of  U.S. 
defense  acquisition,  characterizes  the  ways  in  which  those  models  fall  short  of  meeting  the 
challenges  of  complex  systems-of-systems  acquisition,  and  offers  preliminary  observations 
on  best  practices  to  overcome  those  challenges  based  on  the  results  of  CSIS  research  to 
date.  That  research  to  date  includes  two  new  case  studies.  The  research  is  continuing 
beyond  this  interim  report.  The  final  report  will  reflect  additional  work  and  incorporate  more 
case  studies. 

Introduction 

In  this  age  of  diverse  and  evolving  security  threats,  the  defense  community  is 
acquiring  weapons,  platforms,  and  systems  with  greater  complexity.  Here,  the  term 
complexity  is  used  to  describe  systems-of-systems  (SoS)  involving  multiple,  interrelated 
elements  that  interact  unpredictably.  As  defense  products  and  capabilities  become  more 
complex,  they  are  stressing  the  structure  necessary  for  the  acquisition  of  defense  SoS.  As  a 
result,  the  acquisition  community  has  encountered  operational  challenges  in  maintaining  a 
sufficient  engineering  and  acquisition  workforce  and  process,  as  well  as  outcome  challenges 
in  acquiring  capabilities  on  cost  and  on  schedule. 

SoS  acquisition  poses  considerable  challenges  that  the  current  Department  of 
Defense  (DoD)  acquisition  governance  structure  was  not  necessarily  designed  to  address. 
Increasingly,  defense  capabilities  must  support  the  needs  of  multiple  users  and  must 
operate  as  horizontally  integrated  systems  incorporating  multiple  individual  platforms  and 
programs.  The  high  degree  of  interoperability  and  collaboration  required  for  these  SoS 
capabilities  necessitates  not  only  advanced  systems  engineering  capabilities  but  also 
advanced  governance.  Because  the  technical  capabilities  needed  to  achieve  national 
defense  missions  have  grown  beyond  the  existing  models  of  governance  used  to  acquire 
them,  the  DoD  faces  challenges  in  developing,  procuring,  and  deploying  next-generation 
weapons  and  platforms.  Furthermore,  cost  growth  in  its  portfolio  of  accounts  demonstrates 
that  the  DoD  is  encountering  challenges  managing  cost  and  schedule  risks  associated  with 
advanced  and  integrated  capabilities. 

The  research  of  the  Center  for  Strategic  and  International  Studies  (CSIS)  on  SoS 
acquisition  governance  best  practices  aims  to  help  the  defense  acquisition  community 
address  some  of  the  challenges  of  complexity  and  improve  its  governance  processes  for 
such  acquisitions.  The  specific  research  covered  in  this  interim  report  is  supported  by  a 
grant  from  the  Fleet  Logistics  Center  of  the  Naval  Supply  Systems  Command,  under  the 
auspices  of  the  Acquisition  Research  Program  of  the  Naval  Postgraduate  School.  The 
research  under  this  project  will  conclude  with  a  final  report  to  be  submitted  in  September 
2013. 


This  report  focuses  on  the  practical  application  of  the  CSIS  eight-attribute 
governance  framework  in  comparative  governance  model  analysis.  Specifically,  it  observes 
how  the  attribute  profile  of  two  case  studies — Future  Combat  Systems  (FCS)  and  Maritime 
Domain  Awareness  (MDA) — has  influenced  the  processes  and  outcomes  of  those 
programs.  It  outlines  the  challenges  of  complexity  in  SoS  acquisition  and  the  historical 
origins  of  those  challenges.  It  discusses  the  eight-attribute  framework  for  the  evaluation  of 
individual  acquisition  programs  and  applies  that  framework  to  the  first  two  example  case 
studies.  Finally,  it  discusses  themes  indicated  by  the  case  study  comparison.  In  the  end,  this 
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analysis  suggests  that  getting  governance  right  is  critical  to  addressing  the  barriers  to 
effective  SoS  acquisition. 

Complexity  and  Acquisition  Governance 

The  growing  complexity  of  defense  systems  is  not  a  new  challenge  to  the  defense 
acquisition  community.  Technology  developments  in  defense  have  often  outpaced  the 
evolution  of  federal  and  defense  procurement  processes,  and  policies  for  acquisition 
governance  have  had  to  keep  pace.  As  Berteau,  Ben-Ari,  and  Zlatnik  (2009)  noted  in  a 
report  by  CSIS,  “Current  approaches  [to  acquisition]  were  developed  years  ago  in  an 
environment  where  the  government  was  technically  astute  and  worked  closely  with  one 
vertically  integrated  contractor  per  program”  (p.  2).  Today,  acquisition  programs  regularly 
involve  large  networks  of  contractors  that  integrate  increasingly  complex  technologies  into 
SoS,  but  the  government  has  not  maintained  the  kind  of  strong  technical  workforce 
necessary  for  it  to  stay  on  top  of  these  emerging  technologies.  This  gap  makes  it  more 
difficult  for  requirement  setters  to  independently  forecast  development  schedules  and 
component  compatibility. 

By  examining  the  applications  of  weapon  system  acquisition  models  over  time,  this 
analysis  provides  a  basis  for  the  CSIS  acquisition  governance  framework  in  this  report  and 
the  application  of  that  framework  to  the  case  studies  presented  as  follows.  The  analysis 
outlines  two  areas  of  foundational  knowledge.  First,  how  have  preferred  models  for 
acquisition  governance  evolved  over  time?  Second,  how  does  growing  complexity  challenge 
the  contemporary  models  for  acquisition  governance? 

Evolution  of  Acquisition  Governance  Models — Historical  Context 

The  DoD  has  exercised  different  approaches  to  acquisition  over  time,  favoring 
various  divisions  of  responsibility  between  industry  suppliers  and  government  customers. 
Harvey  Sapolsky  (2009)  outlined  these  models  in  a  paper  published  by  CSIS,  titled  Models 
for  Governing  Large  Systems  Projects.  Sapolsky’s  discussion  suggests  that  the  government 
has  preferred  to  push  more  of  the  functional  responsibilities  of  acquisition  away  from  itself 
and  toward  industry  contractors  over  time.  This  is  in  part  a  product  of  the  flow  of  human 
capital  toward  the  private  sector  and  the  erosion  of  the  government’s  internal  engineering 
expertise  relative  to  industry.  While  the  elements  of  the  Sapolsky  (2009)  model  have 
different  levels  of  analytic  validation,  the  overall  trends  of  skill  and  task  migration  from 
government  to  industry  are  well-documented  and  difficult  to  dispute. 

The  original  model  for  weapons  acquisition  dates  back  to  the  earliest  days  of  the 
U.S.  defense  infrastructure.  At  that  time,  the  U.S.  Navy  could  specify  the  warship  it  needed 
along  with  the  design,  construction,  and  outfitting  of  the  ship.  The  Navy  managed  and 
performed  production  operations  and  generated  technical  requirements  at  all  levels  of  the 
acquisition  chain.  Sapolsky  (2009)  titled  this  acquisition  approach  the  “Arsenal  Model,” 
under  which  the  government  forms  its  own  industrial  base.  It  relies  on  scientists  and 
engineers  within  the  federal  government’s  defense  workforce.  It  is  still  employed  to  an 
extent  today  through  the  DoD’s  network  of  arsenals  and  maintenance  depots  around  the 
country. 

An  acquisition  approach  known  as  the  “Contract  Model”  involves  greater  industry 
participation  in  technical  execution  than  the  Arsenal  Model.  This  model  became  dominant 
with  the  beginning  of  the  Cold  War.  Increasingly,  the  government  relied  on  the  expertise  and 
responsiveness  of  contractors  to  meet  its  needs  for  larger  and  more  technically  demanding 
weapon  systems.  Overtime,  the  government  maintained  a  workforce  in  contracting  and 
acquisition  program  governance  but  began  to  outsource  more  technical  execution  to 
industry. 
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As  weapons  became  more  complex  and  management  of  these  systems  needed 
improvement,  the  acquisition  community  developed  a  preference  for  greater  industry 
involvement  under  the  Weapon  System  Manager  Model  of  acquisition.  This  model  employs 
large  contractors  responsible  for  the  administration  and  coordination  of  a  network  of 
contractors  working  on  subtasks  integral  to  the  overall  acquisition  effort.  Passing 
responsibility  to  the  weapon  system  managers  has  the  advantage  of  involving  large  and 
responsive  contractors  that  assist  the  integration  of  more  complex  systems  that  originate 
from  a  larger  network  of  stakeholders. 

As  the  DoD  began  to  manage  less  of  the  implementation  and  program  management 
capabilities,  it  also  started  to  lose  its  ability  to  provide  technical  direction  for  its  acquisition 
efforts.  This  was  accelerated  by  the  end  of  the  Cold  War,  when  technical  direction  almost 
exclusively  became  the  purview  of  industry.  At  this  time,  DoD  leadership  preserved  combat 
capabilities  while  seeking  savings  within  the  technical  functions  of  the  services.  This  fourth 
model,  known  as  the  Outsourcing  Model,  grew  more  prevalent  due  to  greater  preference  for 
private-sector  program  implementation  over  government  implementation. 

The  flow  of  more  tasks  and  responsibility  toward  industry  contributed  to  the  growth  of 
what  Sapolsky  (2009)  called  the  “Lead  System  Integrator  (LSI)  Model”  more  commonly  used 
today.  Because  the  LSI  Model  has  been  adopted  to  describe  a  specific  type  of  contracting, 
this  paper  refers  to  Sapolsky’s  LSI  Model  as  the  “System  Integration  (SI)  Model.”  In  the  SI 
Model,  capabilities  requirements  are  still  controlled  by  military  officers,  but  technical 
expertise  is  contracted  to  Sis  to  advance  the  capabilities  of  the  planned  weapon  systems. 

As  the  adaptation  of  Sapolsky’s  (2009)  governance  models  in  Table  1  indicates,  the 
evolution  of  acquisition  governance  models  over  time — from  preference  toward  the  Arsenal 
Model  in  the  earliest  days  of  U.S.  defense  acquisition  to  greater  use  of  the  SI  Model  in  large 
weapon  systems  acquisition  today — is  characterized  by  the  gradual  removal  of  responsibility 
from  the  government  buyer.  In  theory,  moving  all  of  the  functions  formerly  performed  by  the 
government  to  industry  contractors  lessens  the  personnel  burden  associated  with  the 
maintenance  of  a  large  in-house  acquisition  workforce.  Furthermore,  reliance  upon  industry 
to  designate  technologies  that  meet  warfighter  demands  should  allow  government  buyers  to 
internalize  Moore’s  Law  and  procure  and  deploy  advanced  capabilities  in  a  shorter  amount 
of  time.  However,  this  evolution  has  fallen  short  of  expectations  in  practice  and  may  instead 
be  contributing  to  cost  and  schedule  overruns  and  compromising  the  government’s  ability  to 
manage  large-scale  acquisition  efforts. 
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Table  1.  Evolution  of  Acquisition  Governance  Models 

(Sapolsky,  2009) 


Model 

Task  ^ 

ARSENAL 

CONTRACT 

WEAPON 

SYSTEM 

MANAGER 

OUTSOURCIN 

G  TO  PRIVATE 
ARSENAL 

SYSTEM 

INTEGRATOR 

PROGRAM 

REQUIREMENTS 

Government 

Government 

Government 

Government 

Government/ 

Industry 

TECHNICAL 

DIRECTION 

Government 

Government 

Government 

Industry 

Industry 

PROGRAM 

MANAGEMENT 

Government 

Government 

Industry 

Industry 

Industry 

TECHNICAL 

EXECUTION 

Government 

Industry 

Industry 

Industry 

Industry 

EXTERNAL 

ENVIRONMENT 

Infrequent  wars 

Little  commercial 
application  of 
military  tech 

Some 
commercial 
application  of 
military  tech 

Private  sector 
pays  better, 
can  be  more 
responsive 

Weapons 
become  more 
complicated 
/complex 

Coordination  of 
sub-systems 
becomes 
important 

Large 

companies  can 
leverage 
political  support 
more  effectively 

Government 
begins  to  lose 
in-house  tech 
capabilities 

Outsourcing 

becomes 

increasingly 

acceptable 

Loss  of  in- 
house 

government 

tech 

capabilities 
leads  to 
inability  to 
define  what’s 
possible 

Note.  The  table  has  been  adapted  by  CSIS  and  also  appears  in  G.  Ben-Ari  and  P.  Chao’s  Organizing  for 
a  Complex  World:  Developing  Tomorrow’s  Defense  and  Net-Centric  Systems  (p.  26). 


The  changing  distribution  of  responsibilities  between  the  industry  supplier  and  the 
government  customer,  reflected  in  the  Sapolsky  (2009)  model,  serves  to  frame  acquisition 
governance  challenges  in  SoS  acquisition.  In  addition,  there  are  two  distinct  models  for  the 
direction  of  acquisition  governance.  In  the  traditional  approach  to  acquisition  governance, 
the  capabilities  comprising  a  SoS  are  governed  downward  from  the  program  level.  In  an 
enterprise-wide  governance  approach,  governance  flows  upward  from  the  capabilities  level. 
Because  the  “enterprise  approach”  is  an  emerging  model  that  is  currently  evolving  to  meet 
the  demands  of  SoS  complexity,  its  application  is  not  evident  in  early-stage  acquisition 
governance  models.  Instead,  traditional,  top-down  approaches  to  acquisition  governance 
have  been  most  prevalent  throughout  the  evolution  of  governance  models,  from  the  Arsenal 
Model  at  the  dawn  of  U.S.  armed  services  to  the  SI  Model  today. 

Complexity — The  Problem  Defined 

The  DoD’s  preferences  for  acquisition  governance  models  have  changed  over  time, 
reflecting  the  flow  of  human  capital,  technical  knowledge,  and  production  assets  away  from 
the  government.  In  the  process,  the  growth  in  weapon  system  complexity  has  outgrown 
existing  models.  Warfighters  are  increasingly  reliant  on  capabilities  developed  as  part  of  a 
broader  SoS  capability.  These  complex  SoS  are  noteworthy  for  their  size,  scope,  and 
inherent  complexity.  In  this  context,  complexity  is  used  to  mean  that  systems  consist  of 
multiple  elements  that  are  typically  developed  and  managed  by  more  than  one  organization. 
This  management  dispersion  adds  to  the  complexity  of  unpredictable  interactions  inherent  in 
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complex  SoS.  Furthermore,  these  system  elements  are  often  part  of  more  than  one  set  of 
capabilities;  that  is,  a  given  system  may  be  part  of  several  SoS.  This  poses  significant 
management  and  governance  challenges.  However,  the  payoff  of  successful  delivery  is  the 
ability  to  achieve  capabilities  far  greater  than  those  delivered  through  individual  programs  or 
systems. 

The  divergence  of  current  governance  models  from  the  service’s  complex  SoS 
capabilities  requirements  is  apparent  in  the  pervasive  acquisition  challenges  that  the  DoD 
faces.  As  capabilities  become  more  complex,  they  demand  a  DoD  systems  engineering 
workforce  that  may  exceed  what  the  government  customer  can  offer.  Additionally,  the  DoD 
faces  operational  requirements  from  the  demand  for  systems  operability  in  multiple  roles, 
missions,  and  environments.  These  challenges  can  result  in  structural  difficulties  for  SoS 
capabilities  that  may  not  exist  in  traditional  acquisition  approaches.  For  instance,  knowledge 
sharing — a  straightforward  task  in  traditional  acquisition — faces  new  challenges  in  complex 
SoS  acquisition  efforts.  Knowledge  ownership  and  incentives  for  sharing  become  less  clear, 
adding  to  the  host  of  governance  process  shortfalls.  Compounding  these  challenges  in 
technology,  operational  requirements,  and  structure,  the  DoD  organizations  needed  to 
develop  and  deploy  the  SoS  capabilities  are  bigger  and  more  difficult  to  manage  and 
maintain  than  traditional  acquisition  organizations,  particularly  under  the  SI  Model. 

The  growing  divide  between  acquisition  governance  models  and  acquisition  in 
practice  is  also  clear  in  SoS  acquisition  outcomes.  The  government  customers’  ability  to 
deliver  complex  SoS  capabilities  on  cost  in  particular  is  declining.  According  to  the 
Government  Accountability  Office  (GAO;  2013),  more  than  86  Major  Defense  Acquisition 
Programs  (MDAPs)  in  fiscal  year  2012  showed  approximately  $400  billion  in  aggregated 
cost  overruns  since  their  first  full-year  estimates,  representing  a  4%  ($90  billion)  growth  in 
development  costs  and  a  5%  ($290  billion)  growth  in  costs  of  procurement.  As  dramatic  as 
this  cost  growth  is,  this  latest  annual  report  from  the  GAO  is  actually  anomalous  when 
compared  against  even  greater  cost  growth  in  the  2012  GAO  report.  In  that  iteration,  96 
MDAPs  existing  in  that  year  had  grown  an  aggregated  $447  billion  in  excess  of  their  original 
estimated  costs  (GAO,  2012).  Given  the  expected  impact  of  sequestration,  the  2012  GAO 
report  is  likely  to  more  accurately  represent  the  trend  in  cost  growth.  That  trend  is 
particularly  evident  when  compared  with  the  2007  GAO  report,  which  cited  64  MDAPs  in  the 
DoD’s  accounts  that  had  grown  at  an  average  annual  rate  of  4.9%.  This  produced  a  total 
annual  cost  growth  of  $165  billion  by  those  programs  in  that  year  (GAO,  2007,  p.  8).  This 
indicates  that  cost  overruns  grew  1 70.9%  in  the  years  between  2006  and  2011. 

The  government  is  also  encountering  challenges  in  keeping  its  major  weapons 
programs  on  schedule.  In  its  latest  report,  the  GAO  (2013)  found  that  MDAPs  experienced 
an  average  delay  of  27  months  in  reaching  initial  operational  capability.  This  figure  exceeds 
the  2012  estimate  of  23  months  in  average  delay  (GAO,  2012,  pp.  6-7).  Combined  with  the 
upward  trend  in  cost  growth  over  time,  this  track  record  indicates  that  existing  governance 
and  management  tools  no  longer  suffice  for  today’s  complex  weapon  systems. 

As  cost  and  schedule  overrun  trends  illustrate,  delivering  SoS  depends  on  getting 
governance  right.  However,  the  traditional  service-centric  approach  to  acquisition 
governance  is  not  sufficiently  flexible  to  meet  the  needs  of  SoS.  Specifically,  flexibility  is 
limited  in  two  ways. 

First,  the  current  process  of  generating  requirements  does  not  allow  for  the 
integration  of  changes  in  user  needs.  Because  complex  SoSs  are  inherently  dynamic,  non¬ 
linear,  and  risk-intensive,  acquisition  leadership’s  ability  to  react  to  changes  in  user  needs  is 
critical  to  the  successful  delivery  of  SoS  capabilities.  Structured  but  flexible  oversight 
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procedures  improve  alignment  between  DoD  requirements  and  fielded  systems  by 
establishing  clear  systems-level  metrics  and  measuring  progress  toward  declared  goals. 
Systems  must  also  be  able  to  respond  to  changes  in  external  factors  in  order  to  ensure  that 
the  SoS  capability  is  as  relevant  when  it  reaches  the  production  and  deployment  phase  as  it 
was  in  pre-acquisition  phases.  These  factors  could  include  macro-level  changes  in  the 
security  environment  and  technological  advances  as  well  as  micro-level  changes  in 
organizational  politics  and  acquisition  effort  leadership. 

Second,  successful  acquisition  delivery  requires  the  “power  of  the  purse”  to  direct 
solutions  and  approaches.  In  order  to  direct  efforts  toward  certain  capabilities,  program 
leadership  must  be  able  to  dedicate  resources  such  as  real  contracting  dollars,  as  well  as 
human  capital  and  allocations  for  other  overhead  costs,  to  certain  system  efforts.  However, 
budgetary  power  is  limited  when  individual  services  and  defense  agencies  are  the  highest 
level  of  governance,  due  to  the  relatively  more  limited  ability  of  those  stakeholders  to 
guarantee  funds  for  the  system  or  to  be  able  to  shift  and  reapportion  them  at  the  system 
level.  The  process  by  which  funds  are  secured  also  limits  flexibility;  the  DoD’s  20-month-plus 
budget  cycle  that  precedes  actual  appropriation  may  lead  the  DoD  to  acquire  technologies 
that  are  bleeding-edge  when  a  budget  is  begun  but  that  may  become  outdated  by  the  time 
the  budget  is  enacted. 

Given  the  limited  effectiveness  of  traditional  service-centric  governance  approaches, 
it  is  useful  to  look  at  new,  enterprise-wide  governance  models  for  the  acquisition  and 
delivery  of  complex  SoS  capabilities.  Numerous  platforms  and  systems  comprise  SoS,  and 
the  interactions  of  these  components  are  highly  unpredictable.  Coordination  of  these  internal 
constituent  systems  is  necessary  to  achieve  the  desired  SoS  capability,  which  otherwise 
would  be  out  of  reach  for  any  single  component  alone.  An  enterprise-wide  approach  to 
governance  would  facilitate  oversight  and  accountability  of  the  systems’  individual 
components  to  achieve  that  coordination.  The  following  case  study  was  selected  to  reflect 
both  the  traditional  and  enterprise  approaches. 

Acquisition  Governance  Case  Studies 

CSIS  is  analyzing  selected  SoS  case  studies  in  order  to  understand  the  merits  and 
demerits  of  existing  governance  models  for  SoS  acquisition.  Once  complete,  and  adding  to 
the  following  two  case  studies,  the  case  studies  will  contribute  to  a  theoretical  framework  for 
the  development  of  a  new  acquisition  governance  model. 

To  date,  the  CSIS  project  team  has  conducted  preliminary  analysis  of  seven  SoS 
case  studies.  Two  relevant  case  studies  are  included  in  this  interim  report.  The  remaining 
case  studies  will  be  added  to  the  final  report.  The  FCS  program  case  presents  an  example 
of  a  so-called  traditional  governance  approach,  in  which  governance  of  all  systems  is 
centralized  at  the  program  level  and  a  program  of  record  is  established  by  the  customer  to 
procure  the  end-user  capability.  The  MDA  effort  provides  an  example  of  an  enterprise-wide 
approach  to  governance,  in  which  families  of  capabilities  are  procured  under  the  umbrella  of 
larger  programs  with  the  MDA  end-user  capability  in  mind.  Together,  the  two  cases  permit 
the  introduction  of  a  CSIS  framework  for  the  identification  and  application  of  acquisition 
governance  attributes.  Next,  this  interim  report  discusses  the  FCS  and  MDA  case  studies  to 
offer  a  high-level  picture  of  program  performance.  Finally,  the  framework  is  used  to  compare 
the  FCS  and  MDA  governance  attributes  and  observe  their  relationship  to  program 
performance. 

Applying  the  CSIS  Acquisition  Governance  Framework  in  Case  Study  Analysis 

Through  previous  work  on  acquisition  models  for  the  Office  of  the  Secretary  of 
Defense  (OSD),  CSIS  has  developed  a  framework  for  analyzing  acquisition  program 
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governance.  The  framework  of  eight  attributes  is  presented  for  public  release  for  the  first 
time  in  Figure  1 . 

This  framework  is  based  on  the  CSIS  process  of  research  on  SoS  governance 
models,  interviews  with  programmatic  stakeholders  and  industry  leaders,  and  findings 
revision  through  input  from  SoS  experts.  The  framework  attributes  represent  concerns, 
questions,  and  issues  that  must  be  addressed  for  an  organization  to  be  able  to  deliver  SoS 
capabilities  effectively.  The  significance  of  these  attributes  varies  depending  on  the  SoS 
being  analyzed,  but  all  eight  attributes  add  important  value  to  the  final  analysis. 


GOVERNANCE  ATTRIBUTE 

DESCRIPTION 

LEVEL  OF 

ORGANIZATIONAL  FOCUS 

The  level  at  which  SoS  governance  occurs  within  the  organization.  This  is  not  the 
same  as  systems'capabilities  focus  or  technical  focus,  both  of  which  are  outside  the 
scope  of  the  CSIS  SoS  governance  analysis. 

INTEGRATION  OF 

FUNCTIONAL  END-USER 
NEEDS 

The  mechanism(s)  and  frequency  with  which  the  functional  needs  of  end-users  are 
built  into  the  system-of-systems.  and  at  which  points  in  the  process  of  delivering  the 
system-of-systems  this  incorporation  occurs. 

DECISION-MAKING 

AUTHORITY 

The  formal  mechanism  by  which  delivery  of  systems-of-systems  is  governed, 
including  how  budget  is  allocated,  standards  are  set,  tradeoffs  are  managed,  and 
inconsistencies  are  adjudicated. 

ENFORCEMENT 

The  mechanisms  and  level  of  management  oversight  by  which  the  objectives  of  the 

SoS  capability  to  be  delivered  are  ensured,  including  measurements  and  program 
metrics. 

WORKFORCE 

The  examination  of  SoS  workforce  structures,  unity  of  mission,  and  capability 
development  and  enhancement  through  use  of  contracted  support. 

INCENTIVE  STRUCTURE 

The  alignment  between  enterprise  goals  and  the  incentive  and  reward  structures 
of  the  components  that  implement  them. 

KNOWLEDGE  OWNERSHIP 
/ACCESS  TO  KNOWLEDGE 

The  accessibility  of  information  regarding  the  operating  environment,  technical 
standards,  and  other  elements  that  comprise  the  system-of-systems. 

RISK  ASSESSMENT  /  RISK 
MANAGEMENT 

Risk  assessments  and  management  strategies  tailored  to  mission  accomplishment 
and  the  flexibility  and  resilience  required  for  delivering  systems-of-systems  in  the 
face  of  unforeseen  developments. 

Figure  1 .  Eight  Attributes  of  Acquisition  Governance 


This  framework  enables  side-by-side  comparison  of  programs  like  FCS  and  MDA.  As 
demonstrated  in  the  following  case  study  comparison,  the  comprehensive  acquisition 
governance  framework  allows  dissection  of  an  acquisition  effort  in  order  to  examine  the 
eight  governance  attributes  independent  of  one  another  and  to  determine  how  the  attributes 
correlate  with  quantitative  performance  metrics  (e.g.,  gaps  between  test  results  and  planned 
goals,  schedule  delays  in  months,  cost  overruns  in  billions  of  dollars,  etc.).  As  part  of  this 
analysis,  CSIS  will  be  able  to  identify  which  attributes  correspond  more  closely  with  poor 
acquisition  governance.  The  analysis  will  also  clarify  and  improve  the  eight-attribute 
framework  outlining  best  practices  in  SoS  acquisition  governance. 

Figure  2  compares  the  FCS  and  MDA  cases  to  illustrate  how  this  methodology  can 
be  applied  in  practice.  The  comparison  offers  a  notional  qualitative  analysis  of  program 
characteristics  for  each  of  the  eight  attributes.  Program  characteristics  are  presented  without 
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direct  comparison  to  one  another  and  independent  of  the  outcomes  and  performance  of  the 
programs  themselves.  Instead,  this  first-level  comparison  serves  to  compose  an  initial  profile 
of  the  programs  for  further  analysis  of  best  practices. 

Future  Combat  Systems  Case  Study 

The  FCS  program  is  one  example  of  how  traditional  acquisition  governance  has 
fallen  short  of  meeting  the  challenges  of  complexity.  This  program,  officially  initiated  with  a 
four-team  competition  in  February  2000  and  terminated  nine  years  later  in  2009,  was  to  be 
the  Army’s  major  research,  development,  and  acquisition  program.  It  initially  consisted  of  18 
manned  and  unmanned  systems  linked  together  via  a  network.  As  the  largest  acquisition 
program  ever  attempted  by  the  Army,  FCS  was  envisioned  to  transform  the  service  by 
replacing  current  systems  such  as  the  M-1  Abrams  tank  and  the  M-2  Bradley  infantry 
fighting  vehicle  as  well  as  by  adding  new  capabilities.  The  advanced  technologies 
associated  with  the  FCS  program,  in  addition  to  the  challenge  of  integrating  subsystems, 
posed  large  problems  for  successful  development  and  contributed  to  the  program’s  high 
level  of  risk.  The  total  cost  for  FCS  ballooned  to  an  estimated  $200  billion  at  the  time  of  its 
cancellation  (Corrin,  2012). 

In  May  2000,  the  Defense  Advanced  Research  Projects  Agency  (DARPA)  awarded 
four  contracts  to  four  industry  teams  to  develop  FCS  designs.  The  Army  awarded  the  LSI 
contract  to  a  team  led  by  Boeing  and  Science  Applications  International  Corporation  (SAIC) 
in  March  2002  after  nearly  two  years  of  design  evaluation.  The  Boeing  and  SAIC  team 
worked  with  more  than  550  contractors  and  subcontractors  in  41  states. 

The  termination  of  FCS  can  be  attributed  to  any  of  a  number  of  problems.  Three 
related  to  program  flexibility  are  worth  discussing  here.  First,  the  capabilities  being 
developed  under  the  program  had  fallen  out  of  alignment  with  warfighter  needs.  The  Army 
began  developing  the  FCS  concept  in  the  1990s,  and  the  service  had  designed  the  system 
to  meet  requirements  for  rapid  deployability,  in-theater  tactical  maneuverability,  and 
survivability.  These  requirements  were  not  suited  to  the  close-combat  and  urban  terrain 
operations  in  which  the  Army  was  engaged  post-9/1 1  (Pernin  et  al.,  2012,  pp.  54-57). 
Second,  several  evolutionary  capabilities  essential  to  program  success  were  not  developed 
at  the  projected  pace.  In  its  attempt  to  avoid  capabilities  obsolescence,  the  Army  chose 
evolutionary  capabilities  to  meet  the  core  survivability  requirements  of  the  manned  ground 
vehicle  (MGV).  The  situational  awareness  and  understanding  (SA/SU)  network,  for  example, 
was  consistently  behind  its  projected  development  schedule  (Pernin  et  al.,  2012,  pp.  109- 
110,  264).  Third,  in  part  due  to  the  slow  pace  of  capabilities  development  and  the  high  rate 
of  expenditure,  remaining  resources  were  low  compared  with  the  level  of  progress  achieved. 
Sources  projected  in  2009  that  60%  of  the  project’s  funding  would  be  exhausted  by  the 
Preliminary  Design  Review,  leaving  the  remainder  to  cover  the  entire  system  development 
phase  (GAO,  2012).  In  the  end,  FCS  was  terminated,  and  the  Army  continued  the 
development  of  some  select  families  of  capabilities  as  individual  systems  (Malenic,  2009, 
2010). 

Maritime  Domain  Awareness  Case  Study 

MDA  provides  a  useful  contrast  with  traditional  acquisition  governance.  MDA  is  not  a 
formal  program  like  FCS  and  other  traditional  acquisition  efforts.  Instead,  this  enterprise 
approach  to  capabilities  acquisition  is  an  interagency,  international  strategy  to  deal  with 
threats  and  challenges  in  maritime  theaters.  The  MDA  system  aims  to  support  and  integrate 
government-wide  efforts  to  collect,  fuse,  analyze,  and  disseminate  data  among  defense,  law 
enforcement,  and  border  protection  officials  from  the  U.S.  and  allied  countries,  creating  a 
cross-domain  common  operating  view. 
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The  MDA  concept  originated  from  a  1998  presidential  initiative  and  was  developed 
further  in  two  presidential  directives  (NSPD-41  and  HSPD-1 3)  released  on  December  21 , 
2004.  Since  then,  the  established  technology  investment  strategy  and  its  supporting  offices 
and  business  systems  have  undergone  numerous  changes.  Currently,  the  National  Maritime 
Intelligence-Integration  Office  (NMIO),  under  the  direction  of  appointees  from  the  Navy  and 
Coast  Guard,  is  the  nominal  lead  for  MDA’s  information  exchange  portal,  the  open 
architecture  tool  at  the  heart  of  the  MDA  mission. 

Two  stages  or  “spirals”  of  technology  acquisition  were  planned  to  develop  and 
integrate  the  infrastructure  that  MDA  would  need  to  achieve  its  mission  objectives.  While  a 
number  of  policy  directives  have  offered  strategic  direction,  the  participating  agencies — 
primarily  the  U.S.  Coast  Guard  and  Navy,  with  some  additional  contributions  from  Customs 
and  Border  Protection — have  mostly  led  their  own  initiatives.  Each  entity  is  free  to  develop 
its  own  definition  of  maritime  domain  awareness.  These  definitions  may  not  fit  together 
across  the  enterprise,  even  though  they  are  generated  from  the  bottom  up.  As  one  former 
MDA  executive  expressed  in  a  conversation  with  CSIS,  “To  some,  binoculars  are  a  maritime 
domain  awareness  technology.” 

Analysis  of  budget  requests  from  2009  to  the  present  day  shows  that  to  date, 
services  have  procured  MDA  assets  mostly  under  the  umbrella  of  larger  program  elements. 
Furthermore,  with  the  exception  of  some  research  and  development  programs  led  by  the 
Office  of  Naval  Research  (ONR),  the  services  have  procured  primarily  non-development 
items  (NDI)  and  commercial  off-the  shelf  (COTS)  products.  As  a  result  of  this  approach  to 
acquisition,  investments  in  MDA  are  difficult  to  quantify,  progress  is  difficult  to  monitor,  and 
oversight  is  difficult  to  ensure. 

Individually,  the  attributes  described  in  Figure  2  underscore  some  merits  and 
demerits  of  both  traditional  and  enterprise-wide  approaches  to  acquisition  governance.  The 
attributes  also  offer  additional  insight  into  acquisition  governance  best  practices  when 
viewed  in  comparison  across  the  two  case  study  programs.  Thus,  there  are  key  attribute- 
specific  takeaways  offered  by  the  case  studies,  both  in  isolation  and  compared  to  one 
another.  These  takeaways  are  discussed  in  Figure  2. 
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GOVERNANCE  ATTRIBUTE  FUTURE  COMBAT  SYSTEMS 

MARITIME  DOMAIN  AWARENESS 

LEVEL  OF 

ORGANIZATIONAL  FOCUS 

•  Focus  on  the  final  product,  governance 
centralized  at  the  program  level 

•  Implementation  at  sub-program  level 

•  Enterprise-level  integration  has  been 
prescribed  but  not  implemented 

INTEGRATION  OF 

FUNCTIONAL  END-USER 

NEEDS 

•  Top-down  requirements-setting  produced 
low  integration  of  end-user  needs  into 
ongoing  FCS  development 

•  MDA  infrastructure  integrates  end-user  needs 
as  they  were  described  at  program  start 

*  Sub-program  funding  allows  for  folding  of 
new  needs  into  spirals  of  technologies 

DECISION-MAKING 

AUTHORITY 

•  Governance  model  was  organized  to 
segregate  decision-making  and  push  it 
downward 

•  Program  executives  choose  whether  and  to 
what  extent  MDA  should  be  incorporated  in 
investment  objectives 

ENFORCEMENT 

•  Other  Transaction  Authority  contract  format 
with  minimal  management  oversight  of 
contractors 

•  No  substantial  enforcement  guiding  MDA- 
re levant  programs  to  desired  end  state 

*  Lacks  funding  and  structure  necessary  for 
cross-service  enforcement  of  MDA  goals 

WORKFORCE 

•  Contractor  program  managers  act  in  the 
role  of  acquisition  leadership 

•  IPTs  co-led  by  one  appointee  from  the  LSI 
and  one  appointee  from  the  Armv 

•  No  investments  in  new  workforce 
development,  relying  instead  on  existing 
acquisition  workforce  at  individual  program 
offices 

INCENTIVE  STRUCTURE 

•  Performance  incentives  based  on 
completion  of  program  events  (e.g.  design 
reviews):  Cost  incentives  based  on 
projected  hfc-cvcle  cost 

•  Program  managers  have  no  incentive  to 
incorporate  MDA  objectives  into 
requirements  generation  or  procurement 
planning 

KNOWLEDGE  OWNERSHIP 
/ACCESS  TO  KNOWLEDGE 

•  Significant  stovepiping  of  knowledge:  Sub¬ 
contractors  hesitant  to  submit  competitive 
information  to  LSI  team 

*  Informal  knowledge  sharing  occurs  through 
white  papers  and  other  forms  of  thought- 
leadership 

RISK  ASSESSMENT  /  RISK 
MANAGEMENT 

•  PM  FCS  and  the  LSI  conducted 
independent  assessments  of  program  health 
periodically  using  an  Earned  Value 
Management  Svstcm  (EVMS) 

*  Lack  of  meaningful  leadership  mechanisms 
with  no  identified  tools  for  management  of 
unforeseen  circumstances 

Figure  2.  Comparison  of  FCS  and  MDA  Acquisition  Governance  Attributes 


The  cases  illustrate  that  the  impact  of  the  level  of  organizational  focus  in  acquisition 
governance  is  dependent  on  how  much  oversight  is  in  place.  In  FCS,  the  concentration  of 
organizational  focus  at  the  program  level  resulted  in  slow  responsiveness  to  changing 
warfighter  needs  and  factional  protection  of  capabilities  families,  such  as  manned  ground 
vehicles.  Rigid  oversight  from  the  top  constrained  the  program’s  progress. 

In  contrast  with  this  organizational  rigidity,  the  MDA  case  demonstrates  that  a  more 
liberal  organizational  focus  in  an  enterprise-wide  governance  approach  also  presents 
challenges.  MDA  capabilities  acquisition  efforts  are  divided  among  numerous  programs,  and 
the  responsibilities  for  acquisition  governance  decisions  are  siloed  within  separate  and 
occasionally  unrelated  programs.  Progress  on  the  MDA  mission  has  been  slow  in  part 
because  of  this  dynamic.  With  greater  oversight  on  the  direction  of  capabilities  development, 
it  is  possible  that  MDA  assets  could  reach  the  theater  more  quickly  and  interact  more  easily. 

Additionally,  the  case  comparison  suggests  that  setting  requirements  and  adjusting 
them  to  the  changing  demands  of  the  end  user  is  difficult  under  top-down  governance, 
whether  the  responsibility  of  governance  falls  to  the  government  or  industry.  As  discussed 
previously,  industry  began  taking  the  requirements  generation  role  from  the  government 
during  the  evolution  of  acquisition  governance  from  the  Weapon  System  Manager  Model  to 
the  Arsenal  Model.  The  reason  for  this  natural  progression  was  that  the  government  no 
longer  had  the  technical  astuteness  to  manage  more  advanced  technical  requirements. 
However,  the  FCS  industry  LSI  also  failed  to  adjust  requirements  properly  and  was  similarly 
hamstrung  in  its  efforts  to  manage  subsystem  capabilities.  Indeed,  the  size  of  the 
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subcontractor  network  and  the  diversity  of  the  systems  and  capabilities  being  acquired 
exposed  the  insufficiencies  of  a  traditional  approach  to  governance. 

The  MDA  effort,  by  comparison,  has  demonstrated  more  agility  in  its  ability  to  adjust 
requirements  to  the  end-user  community’s  changing  demands.  By  distributing 
responsibilities  for  generating  requirements  across  a  network  of  programs  and  subprograms, 
the  MDA  effort  has  enabled  those  closest  to  the  end  user  to  determine  capabilities 
requirements.  While  guidance  from  the  DoD  Executive  Agent  for  Maritime  Awareness  has 
provided  a  framework  for  the  capabilities  required  in  its  spirals  of  technology,  the 
participating  offices  are  free  to  determine  what  specific  systems  fit  into  that  framework.  The 
flaw  in  this  approach  is  that  the  definition  of  maritime  domain  awareness  per  se  can  differ 
among  the  offices.  Furthermore,  there  are  no  mechanisms  to  ensure  that  tasks  are 
delegated  between  government  and  industry  stakeholders  in  a  way  that  enables  timely  and 
cost-efficient  delivery  of  systems  and  subsystems.  A  critical  system  could  be  delayed  due  to 
a  unit-level  acquisition  effort  being  governed  under  the  LSI  Model,  for  example,  with 
potentially  damaging  consequences  for  the  broader  SoS. 

Thus,  these  differences  in  the  two  attributes  for  Level  of  Organizational  Focus  and 
Integration  of  Functional  End-User  Needs  help  to  identify  challenges  in  both  the  traditional 
and  enterprise-wide  approaches  to  acquisition  governance.  In  contrast,  similarities  between 
FCS  and  MDA  illustrate  that  some  governance  attributes  transcend  the  dimension  of  task 
delegation  between  the  government  and  industry  stakeholder  communities.  Specifically,  a 
comparison  of  the  attributes  for  Enforcement,  Incentive  Structure,  and  Knowledge 
Ownership  shows  that  there  are  critical  oversight  functions  that  both  industry  and 
government  need  to  get  right  in  order  to  facilitate  complex  SoS  acquisition. 

The  FCS  and  MDA  cases  both  illustrate  that  system  delivery  suffers  when  proper 
oversight  enforcement  mechanisms  are  not  in  place.  Use  of  an  Other  Transaction  Authority 
(OTA)  contract  in  FCS  created  oversight  challenges  at  the  systems  level,  resulting  in  limited 
enforcement  of  cost  and  technical  readiness  standards.  It  is  also  reported  that  the  LSI 
issued  contracts  with  standards  below  those  in  its  contract  with  the  Army.  The  Army 
addressed  these  issues  when  it  revised  the  contract  to  a  Federal  Acquisition  Regulation 
(FAR)  contract  and  established  processes  for  greater  Army  involvement  in  capabilities 
selection  and  requirements. 

Enforcement  has  been  lacking  at  both  the  systems  and  SoS  level  within  MDA. 
Because  the  MDA  effort  is  a  loose  network  of  individual  program  offices  guided  by 
overarching  objectives  without  prescriptive  benchmarks,  the  DoD  has  encountered  a  good 
deal  of  difficulty  in  moving  it  forward  and  gauging  its  progress. 

Effective  incentives  for  capability  delivery  on  schedule  and  on  cost  were  also 
noticeably  absent  in  both  programs.  In  the  case  of  FCS,  performance  incentives  were  based 
on  the  completion  of  program  events  such  as  design  reviews,  rather  than  on  a  meaningful 
metrics-based  assessment  of  technical  performance.  The  misalignment  of  technical 
requirements  between  the  LSI  team  and  the  government  compounded  this  problem, 
detracting  from  the  effectiveness  of  incentives  in  encouraging  on-schedule  capability 
delivery.  Cost  incentives  also  were  based  on  artificial  benchmarks,  assessing  contractor 
cost-performance  based  on  the  projected  life  cycle  cost  of  capabilities  rather  than  on  their 
actual  cost  track  record. 

The  patchwork  of  incentives  in  MDA  also  appears  to  be  ineffective.  In  contrast  with 
the  artificial  incentives  in  FCS,  MDA  has  few  incentives  at  all  on  the  SoS  level.  On  the 
systems  level,  individual  programs  are  responsible  for  the  establishment  of  incentives  for  the 
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delivery  of  individual  capabilities.  A  more  explicit  incentives  infrastructure  may  be  necessary 
to  encourage  greater  commitment  to  the  MDA  objectives. 

Finally,  the  FCS  and  MDA  cases  underscore  the  importance  of  knowledge  ownership 
in  delivering  SoS  capabilities.  Significant  stovepiping  was  apparent  in  the  FCS  case,  where 
the  use  of  an  industry-led  acquisition  workforce  raised  concerns  among  subcontractors 
about  the  fairness  of  competition  and  the  use  of  proprietary  information.  These  concerns 
created  barriers  to  transparent  information  sharing  across  and  within  programs.  While 
stovepiping  has  not  been  prevalent  in  MDA,  unstructured  information-sharing  processes, 
combined  with  a  lack  of  uniform  maritime  domain  awareness  definitions,  have  hampered 
knowledge  sharing  among  the  MDA  stakeholders. 

Toward  a  New  Acquisition  Governance  Model 

The  case  study  analysis  presented  previously  offers  several  preliminary  key  findings 
for  the  CSIS  effort  to  identify  governance  best  practices  in  SoS  acquisition.  The  cases  begin 
to  support  three  themes  in  the  creation  of  a  new  governance  model. 

First,  SoS  acquisition  governance  benefits  from  having  strong  and  structured 
management  oversight  procedures  tied  to  documented  results.  Shared  FCS  and  MDA 
shortcomings  in  Enforcement  and  Incentive  Structure  originated  in  insufficient  or  otherwise 
flawed  oversight.  The  reported  inadequacies  of  government  oversight  over  the  LSI 
exacerbated  technical  and  programmatic  flaws  in  FCS.  In  MDA,  minimal  oversight  over  a 
broad  network  of  government  customers  has  caused  problems  with  both  the  establishment 
of  requirements  and  the  measurement  of  progress. 

Second,  existing  approaches  to  acquisition  are  challenged  by  high  barriers  to 
knowledge  sharing  and  a  lack  of  clarity  regarding  the  ownership  of  this  knowledge.  The  use 
of  an  LSI  contractor  in  FCS  created  disincentives  to  information  sharing,  both  within  the 
program  and  with  the  customer.  Industry  stakeholders  perceived  that  information  sharing 
between  and  among  their  peers  would  damage  their  competitiveness  and  compromise  their 
proprietary  information.  While  these  structural  impediments  to  knowledge  sharing  are  not 
apparent  in  MDA,  that  program  instead  has  minimal  incentives  to  promote  knowledge 
sharing.  In  other  words,  explicit  barriers  do  not  exist  between  and  among  MDA 
stakeholders,  but  there  is  insufficient  payoff  for  knowledge  sharing  outside  of  lower  level 
systems. 

Third,  a  new  model  of  acquisition  governance  may  require  more  task  sharing 
between  government  and  industry  than  has  been  seen  in  prior  models.  Some  problems  with 
FCS  are  attributable  to  that  program’s  use  of  an  SI  Model  of  acquisition  governance. 
However,  similar  problems  apparent  in  the  MDA  program  show  that  designating  other  tasks 
to  government — in  the  case  of  that  program,  the  task  of  Program  Management — is  not 
sufficient  to  fix  problems  of  management  oversight.  In  fact,  a  comparison  of  MDA  and  FCS 
shows  that  replacing  industry  with  government  in  non-Program  Requirements  tasks  of  the  SI 
Model  to  create  a  new  model  seems  to  have  created  additional  problems  with  the  lead 
agency’s  ability  to  coordinate  and  enforce  progress  among  other  participating  agencies. 

Conclusion 

As  government  demands  to  defeat  threats  to  global  security  cause  systems  of 
capabilities  to  grow  in  size  and  complexity,  the  limited  effectiveness  of  existing  models  of 
acquisition  governance  is  becoming  more  apparent.  The  initial  analysis  reflected  in  this 
interim  report  underscores  the  importance  of  further  research  on  best  practices  in  acquisition 
governance  with  a  view  toward  the  creation  of  a  new  model.  The  example  case  studies 
presented  here  are  a  first  step  in  an  effort  by  CSIS  to  identify  those  best  practices. 
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The  case  studies  illustrate  that  getting  governance  right  would  involve  a  thorough 
analysis  of  how  program  outcomes  are  affected  by  a  comprehensive  set  of  governance 
attributes.  Each  of  the  eight  attributes  used  here  represents  a  different  but  vital  element  of 
any  acquisition  effort.  Moving  forward,  the  CSIS  Complexity  project  team  will  perform  an 
analysis  similar  to  the  one  discussed  previously  across  a  total  of  seven  SoS  acquisition  case 
studies  to  identify  what  practices  in  acquisition  governance  contribute  to  program  success 
and  what  practices  make  successful  capabilities  development  and  acquisition  more  difficult 
to  achieve.  The  preliminary  cases  presented  here  will  also  be  refined  based  on  the  results  of 
an  ongoing  primary  research  effort. 

This  interim  report  serves  as  a  progress  report  on  efforts  by  CSIS  to  identify  best 
practices  in  SoS  acquisition  governance.  This  effort  will  incorporate  additional  case  studies 
in  the  coming  months  and  will  conclude  in  September  2013  with  a  full  report  on  best 
practices  and  a  framework  for  new  governance  models.  The  report  will  be  presented  at  the 
May  2014  Naval  Postgraduate  School  Acquisition  Research  Symposium. 
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Abstract 

The  complexity  of  developing  and  acquiring  weapons  systems  continues  to  increase  due  to 
highly  integrated  system  architectures,  rapid  technology  evolution,  and  emergence  of  highly 
diverse  set  of  missions.  The  imperatives  of  system-of-systems  (SoS)  integration  and 
interoperability  (l&l)  further  complicate  the  system  acquisition  process.  These  challenges 
continue  to  frustrate  completing  the  acquisition  of  systems  within  time  and  budget  goals. 

The  DoD  has  commonly  assigned  the  role  of  the  lead  system  integrator  (LSI)  to  a  prime 
contractor.  This  is  fraught  with  many  issues  related  to  conflict  of  interest,  performance,  and 
defining  clear  roles  and  responsibilities  (especially  the  inherent  role  of  government).  The  DoD 
has  indicated  that,  in  some  cases,  the  LSI  responsibilities  should  migrate  back  to  the  DoD. 

In  this  paper,  we  discuss  the  roles  of  the  LSI,  where  DoD  acquisition  skills  may  need  to  be 
strengthened  to  perform  as  the  LSI,  and  discuss  methods  and  tools  to  do  so.  This  paper  is  a 
result  of  multi-year  discussions  and  research  with  a  major  Naval  Systems  Command  to  find  a 
path  to  faster  time-to-market  and  higher  levels  of  interoperability  and  integration  of  our 
weapons  system  acquisitions. 
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Introduction 

What  is  a  lead  system  integrator  (LSI)?  Although  a  broader  concept  than  its  simple 
acronym,  commonly,  the  role  of  the  LSI  has  been  turned  over  to  industry  in  the  form  of  a 
prime  contractor,  or  team  of  contractors,  hired  by  the  federal  government  to  execute  a  large, 
complex,  system-of-systems  (SoS),  defense-related  acquisition  programs  (Grasso,  2007). 
The  need  for  an  LSI  is  often  associated  with  the  acquisition  of  an  SoS  or  a  constituent 
system  to  an  SoS.  SoS  programs  acquire  a  collection  of  various  platforms  (e.g.,  ground 
vehicles,  aircraft,  and  ships)  that  are  to  be  linked  together  so  as  to  create  a  larger, 
integrated  overall  system  (Lane,  2006). 

LSIs  are  further  categorized  based  on  their  system  development  capabilities  and 
responsibilities.  Section  805  of  the  National  Defense  Authorization  Act  of  Fiscal  Year  2006 
(2005)  defines  these  two  types  of  LSIs  as 

1 .  “Lead  system  integrators  with  system  responsibility”  prime  contractors  who 
develop  major  systems  that  are  not  expected  at  the  time  of  the  contract 
award,  as  determined  by  the  Secretary  of  Defense,  to  perform  a  substantial 
portion  of  the  work  on  the  system  and  major  subsystems. 

2.  “Lead  system  integrators  without  system  responsibility” — contractors  who 
perform  acquisition  functions  that  are  closely  associated  with  inherently 
governmental  functions  in  the  development  of  a  major  system.  LSIs, 
regardless  of  type,  are  subject  to  the  same  rules  as  other  federal  contractors. 

In  recent  years,  the  LSI  responsibility  has  been  awarded  to  industry  for  major  DoD 
acquisitions.  However,  this  has  led  to  conflict-of-interest  complications  resulting  in  revised 
law  stating,  “No  entity  performing  lead  systems  integrator  functions  in  the  acquisition  of  a 
major  system  by  DoD  may  have  any  direct  financial  interest  in  the  development  or 
construction  of  any  individual  system  or  element  of  any  system  of  systems”  (Defense 
Authorization  Act  for  Fiscal  Year  2006,  Section  807).  As  a  result,  several  of  the  major 
contractors  have  divested  into  companies  focused  separately  on  systems  integration  and 
product  development.  An  example  is  Lockheed  Martin,  where  “Lockheed  Martin’s  decision 
to  divest  the  business  was  based  on  the  U.S.  Government's  increased  concerns  regarding 
perceived  conflicts  of  interest”  (The  SI  Organization,  2010). 

Due  to  recent  failures  in  some  major  DoD  acquisition  programs  (examples  in  GAO, 
2007),  the  DoD  has  made  the  decision  to  use  an  LSI  endure  more  scrutiny  by,  in  some 
cases,  requiring  certification  by  the  Committees  on  Armed  Services  for  both  the  Senate  and 
the  House  (OSD,  2007).  This  has  led  some  to  conclude  that  to  reduce  complexities  and 
risks  associated  with  the  use  of  contractors  as  the  LSIs,  the  DoD  should  consider  (Grasso, 
2007): 

•  prohibiting  the  use  of  private-sector  LSIs  in  future  acquisition  programs; 

•  reducing  the  possible  need  for  private-sector  LSIs  by  building  the  defense 
civilian  and  military  acquisition  workforces  back  up,  and  having  the  DoD 
assume  the  role  of  the  LSI,  and  requiring  that  DoD  manage  all  SOS 
programs;  and 

•  implementing  the  recommendations  of  the  Gansler  Commission  on  improving 
the  acquisition  workforce  (U.S.  Army,  2007). 

The  following  discussion  begins  with  the  premise  that  the  DoD  concurs  with  the 
recommendations  above  and  desires  to  bring  more  LSI  responsibilities  “in  house,”  in 
particular,  engineering  responsibilities.  Some  of  the  major  systems  commands  are  exploring 
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such  a  transformation  to  bring  systems  to  the  DoD  more  quickly  while  attaining  higher  levels 
of  interoperability  (Young,  2010).  If  the  DoD  acquisition  community  wants  to  make  such  a 
transformation  to  retain  more  inherently  governmental  responsibilities  for  major  system 
acquisitions,  what  needs  to  be  done  to  fortify  the  systems  engineering  (SE)  workforce  skills, 
SE  methods,  and  SE  tools  to  enable  taking  on  the  larger  role  of  the  LSI?  We  use  Figure  1 
as  a  context  diagram  throughout  (blue  is  our  focus). 

DoD  Acquisition 
Cycle 

LSI  Management  . 

_  ,  LSI  SE  Roles 

Roles 

I 

SE  Processes 


SDEA  Methods 
and  Tools 


Figure  1.  LSI  Systems  Engineering  and  Management  Roles  Are  Supported 
by  Systems  Engineering  Processes,  Methods,  and  Tools 

In  our  previous  research  (Montgomery,  Carlson,  &  Quartuccio,  2012),  we  focused  on 
how  SE  tools  could  be  applied  to  DoD  acquisition  SE  methods.  We  introduced  a  model- 
base,  SE-inspired  approach  named  System  Definition-Enabled  Acquisition  (SDEA)  in  that 
research  and  discussed  how  SDEA  could  be  instrumental  for  LSI  SE  success.  What  follows 
extends  that  previous  SE  tools  perspective  to  the  role  of  the  LSI  SE  as  a  result  of  ongoing 
research  with  Naval  Air  Systems  Command  (NAVAIR). 

Problem  Definition  and  Research  Questions 

The  Defense  Authorization  Act  for  Fiscal  Year  2006  (Section  807)  provides  emphasis 
on  the  importance  that  the  lead  systems  engineer  on  a  DoD  acquisition  be  an  experienced 
government  employee.  Additionally,  the  DoD  ASD(R&E)  chief  systems  engineer,  Stephen 
Welby  (2012)  summarized  the  imperatives  for  DoD  SE  as  follows: 

As  the  complexity  of  our  systems  has  increased,  so  has  the  need  for  effective 
systems  engineering  throughout  the  life  cycle.  We  face  challenges  in 
implementing  robust  systems  engineering  processes,  from  requirements 
identification  and  analysis  through  technology  and  architecture  selection  and 
assessment,  analysis  and  coordination  of  complex  system  design, 
development,  and  execution  ....  We  are  now  increasingly  focused  on 
addressing  early-acquisition  phases  including  requirements  definition, 
development  planning,  and  early  acquisition  systems  engineering  support. 

Finally,  as  stated  in  a  report  sponsored  by  Welby  (Systems  Engineer  Research 
Center  [SERC],  2010),  “existing  systems  engineering  tools,  processes,  and  technologies 
poorly  support  rapid  design  changes  or  capability  enhancements  within  acceptable  cost  and 
schedule  constraints.” 

Problem  Statement 

The  background  and  guidance  presented  in  the  previous  section  leads  to  our 
investigation,  focused  by  the  following  problem  statement:  The  DoD  does  not  have 
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adequate  SE  methods,  processes,  workflows,  and/or  tools  that  support  the  expansive  role  of 
the  LSI  in  major  weapons  systems  acquisitions. 

Research  Questions 

The  associated  research  questions  that  we  have  been  investigating  in  order  to 
resolve  our  problem  statement  include  the  following: 


•  What  are  key  DoD  acquisition  challenges  for  the  LSI? 

•  What  are  the  key  LSI  roles  and  attributes? 

•  What  is  the  current  state  of  DoD  LSI  maturity? 

•  What  SE  methods  are  prime  candidates  to  improve  upon  to  support  LSI? 

•  How  can  MBSE/SDEA  be  applied  to  the  LSI? 

LSI  Challenges 

Regardless  of  the  government/contractor  ownership  of  LSI  SE  responsibilities,  the 
challenges  to  current  acquisitions  are  diverse,  not  necessarily  new,  and  are  discussed  as 
follows  (derived  from  Montgomery  et  al.,  2012). 


Complex  System  Acquisition 

The  current  DoD  acquisition  process  (see  Figure  2),  as  specified  in  DoD  5000, 
WSARA,  and  a  long  heritage  of  acquisition  experience,  is  based  on  the  acquisition  of  stand¬ 
alone  systems.  Today’s  system  acquisitions  are  more  co-dependent  on  the  development  of 
other  complex  systems  in  an  SoS  environment.  This  requires  a  higher  level  of  coupling 
between  system  engineering  and  the  acquisition  process  to  support  SoS,  as  well  as  the 
need  for  higher  levels  of  LSI  support. 


DoD  Acquisition 
Cycle 


Concept  Development  & 

Validation 

Design  Development  & 

Validation 

Produce  &Qualify 

Deploy  &  Improve 


Figure  2.  DoD  5000  Acquisition  Process 


A  problem  that  continues  to  frustrate  this  acquisition  timeline  and  increase  program 
costs  is  both  system  complexity  and  SoS  interoperability.  Many  acquisitions  are  the 
integration  of  several  systems  that  are  being  acquired  and  developed  independently  and  for 
their  own  purposes.  This  SoS  method  presents  a  new  and  interesting  level  of  complexity  for 
system  engineers  because  system  engineers  rarely  have  the  opportunity  to  affect  the  design 
of  these  co-dependent  systems.  The  functionality,  interfaces,  operational  objectives,  and 
intended  system  environments  all  provide  a  challenge  to  ensuring  that  the  SoS  can  be 
integrated  successfully  while  producing  new  emergent  behaviors  that  are  predictable  and 
satisfy  the  user  needs.  Couple  all  of  this  complexity  and  SoS  realities  to  the  existing  system 
engineering  methods,  practices,  principles,  organization  behaviors,  and  workforce  skills,  and 
the  need  for  SDEA  methods  and  tools  becomes  clearer  to  resolve  many  of  the  following 
challenges. 
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Acquisition  Timeliness 

Acquisitions  are  too  slow-to-market.  Acquisition  schedules  are  often  document- 
driven  and  technical  review-driven  processes  and  non-adaptive  to  changing  or  emergent 
requirements.  DoD  5000  emphasizes  prototypes  early  in  acquisition,  requiring  a  tightly 
coupled  engineering  system  to  meet  engineering  goals,  objectives,  and  requirements.  The 
LSI  SE  needs  to  be  diligent  to  ensure  pre-planned  programmed  improvements  (P3ls)  are 
enabled  and  that  tools  provide  enduring  design  data. 

Acquisition  Process 

The  Acquisition  process  is  not  LSI  design-driven.  The  DoD  5000  acquisition  process 
is  oversight-driven  and  document-driven  and  designed  such  that  government  engineers 
provide  the  oversight  while  the  contractors  provide  the  content.  It  is  likely  that  DoD  5000  will 
ultimately  have  to  be  revised  to  define  a  process  more  aligned  to  the  government  providing 
LSI  SE  direction.  An  example  would  be  to  exploit  a  process  that  that  could  be  streamlined 
as  a  result  of  the  government  and  user  community  retaining  more  LSI  SE  activities  and 
direct  engagement  with  the  development  of  the  baseline  in  lieu  of  merely  reviewing  the 
progress  of  the  contract. 

System  Complexity 

LSI  engineering  capabilities  do  not  always  support  design  and  acquisition  of  highly 
complex  systems.  Simple  systems  and  complex  systems  proceed  through  the  acquisition 
process  essentially  the  same.  The  role  of  the  LSI,  however,  is  more  applicable  for  the  needs 
of  complex  systems  with  a  significant  emphasis  on  defining  the  interaction  of  systems  along 
with  robustness  of  the  system  solution.  This  will  require  a  dramatically  different  way  of 
defining  the  LSI  engineering  process  and  how  it  integrates  with  the  program  management 
processes.  An  example  is  employment  of  tools  and  methods  to  provide  the  ability  to  assess 
SoS  performance  and  emergent  system  behaviors  in  a  quantifiable  manner.  Currently,  the 
ability  to  predict,  manage,  and  control  such  emergent  behaviors  can  be  elusive. 

Integration  and  Interoperability 

Systems  often  fail  at  integration  or  do  not  interoperate  effectively.  Successful 
integration  of  systems,  especially  SoS,  is  challenged  by  functional  gaps  and  overlaps 
among  the  systems’  complex  interfaces  and  a  large  number  of  internal  and  external  system 
interfaces.  SoS  integration  also  demands  the  interoperability  among  these  systems,  as  well 
as  the  interoperability  outside  of  the  system  for  other  systems  that  are  codependent.  The 
LSI  needs  SE  tools  and  methods  that  define  and  manage  risks  associated  with  these  critical 
functions  and  interfaces. 

Total  Ownership  Costs 

Prediction  and  control  of  total  ownership  costs  (TOC)  is  difficult.  The  acquisition  cost 
incurred  during  the  development  cycle  is  only  a  fraction  of  the  total  ownership  cost  of  any 
system.  The  LSI  needs  to  have  very  detailed,  predictable,  and  repeatable  behavior  modeling 
of  both  the  acquired  system  and  external  systems  in  order  to  accurately  predict  and  control 
TOC. 

Engineering  Workforce 

The  veteran  engineers  are  rapidly  retiring  and  not  being  replaced  with  engineers  with 
commensurate  experience.  The  system  design  process  and  SE  tools  need  to  provide  high 
levels  of  repeatability  and  quantifiability  that  is  less  dependent  on  engineering  judgment  and 
more  dependent  on  metrics  that  provide  a  highly  refined  engineering  solution.  Given  that 
many  veteran  engineers  are  retiring,  there  is  a  need  to  provide  system  design-driven 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  657- 


methods  to  a  younger  engineering  community.  A  system  is  needed  that  also  provides 
project-to-project  consistency  and  repeatability. 

Systems  Engineering  Attributes  and  Roles  of  an  LSI 

LSI  Attributes 

Not  all  DoD  acquisitions  need  to  be  managed  by  an  LSI.  Many  systems  can  be 
acquired  with  small  teams  where  complexity  and  risk  are  relatively  easy  to  manage.  The 
following  is  a  list  that  includes  attributes  of  program  and  system  designs  where  the  need  for 
an  LSI  may  be  more  imperative  (partially  derived  from  Loudin,  2010): 

•  Program  importance  and  span  of  impact — high  risk,  large  cost,  or  expansive 
interoperability  impacts  to  the  enterprise 

•  System  or  SoS  complexity — large-scale  complexity  with  a  large  number  of 
high-consequence  risks,  external  SoS  interfaces  and  interactions,  and  high 
likelihood  of  unanticipated,  negative  emergent  behaviors 

•  Stakeholder  relationships — Collaborative  versus  command-and-control 
contractor/government/fleet  user  interactions  are  necessary 

•  Organization  agility  is  required  to  organize  around  acquisition  (versus  the 
obverse) 

•  DoD  determines  that  ownership  of  critical  data  and/or  DoD  reuse  of  critical  IP 
is  mandated 

•  “National  teaming”  is  required  to  ensure  enterprise-level  SoS  issues  are 
intrinsic  to  system  success 

•  Acknowledgement  and  acceptance  of  higher  system  design  and  acquisition 
risks 

•  Rapid  identification  and  adaptation  of  emergent  opportunities  are  essential 

•  Strong  integration  leadership  and  control  is  required 

•  Low  barriers  to  entry  for  technology  and  innovation  need  to  be  established 
and  maintained  throughout  the  life  cycle 

•  Disciplined  and  rigorous  standards  demanded  for  integration  of  other  systems 
into  the  enterprise 

LSI  Roles 

The  roles  of  the  LSI  are  similar  to  the  roles  of  any  SE  or  system  integrator  (SI).  The 
primary  difference  is  the  span  of  design  and  integration  authority  that  persists  throughout  the 
system  acquisition  and/or  complete  life  cycle.  The  following  are  a  sampling  of  the  LSI  roles 
that  are  more  expansive  than  traditional  SE/SI: 

•  Design:  Act  as  the  primary  designer  (sometime  referred  to  as  the  “design 
agent”).  Design  includes  system  and  SoS  designs.  Roles  include  conceptual 
design,  architectural  design  (operational,  functional,  physical,  interface, 
qualification),  and  integration  and  qualification  designs. 

•  Source  selection:  Responsible  for  providing  solicitation  packages,  reviewing 
and  evaluating  proposals,  and  selecting  and  awarding  the  contract  to 
component,  subsystem,  system,  or  product  provider.  Component-level 
solicitation  has  often  been  assigned  to  prime  contractors. 
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•  Subcontractor  selection:  Survey,  vetting,  and  selection  of  providers  of 
components  or  services.  Component-level  selection  has  often  been  assigned 
to  prime  contractors. 

•  Supplier  chain  management:  Engagement  within  the  domains  of  hardware 
and  software  configuration  item  selection,  sources  of  supply,  and 
manufacture. 

•  Trade-off  studies:  Conduct  of  objective  trade-off  studies  and  analysis  of 
system  challenges,  risks,  and  opportunities. 

•  System  baseline  management:  Definition,  control,  and  management  of 
system  design  baselines,  configuration  management,  and  realized 
configurations. 

•  Rigorous,  multi-system  definition  and  management  of  interfaces,  taxonomy, 
system  structures,  and  so  forth. 

•  Coordinator  (and  funder)  of  contributing  research. 

•  End-to-end  span  of  authority  and  control  for  baseline  control  and 
management  of  the  system  design,  development,  integration,  qualification, 
and  deployment. 

•  Qualification  (“V&V”):  Ultimate  responsibility  for  developmental  (verification), 
operational  (validation),  and  acceptance  qualification  success. 

•  Sustainment/suitability:  Responsible  for  sustainment  and  suitability  design  of 
the  system  and  impact  analysis  of  SoS  sustainment  strategies. 

Current  State  of  the  DoD  LSI 

How  do  many  engineering  organizations  operate  today  in  DoD  acquisition?  There 
have  been  many  strains  on  DoD  manpower  reductions  over  recent  years,  and  the  result  has 
been  to  depend  heavily  on  contractors  to  do  the  “heavy  lifting”  in  many  engineering 
domains.  Although  the  government  retains  many  subject-matter  experts  (SMEs),  these 
highly  skilled  staff  are  senior,  retiring  at  a  rapid  rate,  and  are  stretched  thin.  The  larger  and 
more  complex  the  project,  the  more  likely  the  government  has  decided  to  use  a  large  prime 
or  LSI  contractor. 

The  different  roles  of  engineering  involvement  are  shown  in  Figure  3,  spanning  from 
performing  the  role  of  the  “buyer”  for  simple  systems  to  the  role  of  “integrator”  when 
acquiring  complex  systems.  (We  put  forth  these  role  titles  just  to  provide  reference;  they  are 
not  formally  accepted  throughout  DoD).  As  can  be  seen,  the  engineering  tasks 
(requirements  engineering,  design,  etc.)  become  more  expansive  as  the  role  approaches 
that  of  the  LSI  (integrator).  The  roles  close  to  red  (bottom  of  the  list)  are  associated  with 
complex  acquisitions.  Our  assumption  is  that  the  government  has  been  performing  in  the 
yellow  band  of  this  chart  during  recent  years.  As  previously  stated,  contractors  have  been 
more  likely  to  be  assigned  the  majority  of  engineering  duties  as  the  systems  became  more 
complex. 
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•  Role  LSI  SE  LSI  Proqram 

■  'Buyer'  Low  -  (GSA.COTS.NDI)  Low  -  purchase  order 

I  'Purchaser'  Med  -  (R  +  purchase)  Low  -  in-house  +  purchase  order 

I  Acquirer’  Med  -  (R  +  prime)  Med  -  prime  contractor 

1  'Architect/Designer'  High  -  (R.D.Y+  SI  +  lead)  High  -  in-house  +  SI  +  prime/lead 

|  Integrator'  High  -  (R.D.I.Q.Y  +  contractors)  High  -  in-house  +  multiple  contractors 

Legend 

R-Requirements  ^Integration 

D=Design/Architect  Q=Qualification 

V-Development  ^Production 

Y=Deployment 

Figure  3.  The  Engineering  Involvement  of  the  DoD  Acquirer  Becomes  More  Expansive 
for  Complex  (“Integrator”  Role)  Systems  as  Compared  to  Simple  (“Buyer” 

Role)  Systems 

Figure  4  depicts  “traditional”  versus  “LSI”  contractor/government  engineering  span  of 
authority  across  the  DoD  acquisition  cycle.  The  top  portion  (a)  is  a  typical  acquisition  cycle 
that  spans  from  system  concept  to  deployment.  The  middle  portion  (b)  indicates  that  the 
traditional  government  levels  of  engineering  effort  are  maximum  at  the  early  and  latter 
stages  of  the  acquisition,  with  the  contractor  design,  production,  and  integration  in  the 
middle.  The  lower  portion  (c)  of  the  diagram  posits  that,  if  the  government  is  the  LSI,  the 
government  performs  more  of  the  design  and  integration  activities,  and  the  contractor  shifts 
to  a  more  “manufacturing”  role.  Although  there  could  be  many  variations  on  this  LSI  theme, 
what  is  important  to  note  is  that  the  area  under  the  curve  represents  the  government  level  of 
effort.  Some  refer  to  his  role  as  the  “design  agent.”  In  the  LSI  case,  this  level  of  effort  is 
more  expansive  than  today  and  requires  new  methods,  practices,  and  tools  to  support  the 
government  engineer. 
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Figure  4.  The  LSI  Roles  for  the  Government — (c)  Compared  to  (b) — Will  Require  Greater 
Methods,  Practices,  and  Tools  to  Achieve  Work  of  the  Increased  “Area  Under 

the  Curve” 


Another  perspective  is  to  try  to  assess  where  the  “maturity”  of  the  government 
engineering  community  (writ  large)  is  today  and  how  it  needs  to  be  enhanced.  Figure  5  puts 
forward  a  non-scientific  assessment  of  where  that  maturity  may  be  (dotted  line).  The  colors 
align  with  Figure  3  and  shows  that  the  current  DoD  acquisition  workforce  performs 
comfortably  as  a  “buyer”  and  “purchaser”,  often  at  the  “acquirer”  level,  but  has  yet  regularly 
perform  at  the  “architect/designer”  or  “integrator/LSI”  role.  The  graph  shows  that,  as  the 
government  makes  the  transition  to  the  upper  right  of  the  graph,  the  engineering  and 
programmatic  span  of  authority  must,  necessarily,  increase  for  the  government  and 
decrease  for  the  contracting  community. 
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Span  of  Design  Authority 

(Concept  -  Requirements  -  Design  -  Integration  -  Deployment) 


“LSI"  is  not  a  unique  role  or 
state  of  acquisition  but,  rather, 
is  the  most  complex  state  of 
DoD  acquisition  engineering 
where  all/most  program 
management  and  engineering 
activities  are  retained  and 
executed  by  the  ‘design 
agent’  (i.e,  the  Government) 


Current  state  of  DoD  skills, 
experience,  processes, 
methods,  and  tools? 


Figure  5.  Increasing  Role  of  Government  Engineering  Toward  “Integrator”  Will  Involve 
Reducing  Span  of  Design  and  Programmatic  Authority  for  the  Contractor 


Systems  Engineering  Methods  Supporting  LSI 

There  are  probably  very  few  new  fundamental  principles  and  essential  activities  that 
are  required  for  the  LSI;  however,  the  depth  and  ownership  of  SE  activities  are  greater  and 
more  enduring.  In  order  for  the  DoD  to  move  to  the  upper  right-hand  corner  of  Figure  5, 
additional  SE  practices,  methods,  and  tools  need  to  be  enhanced.  A  representative  SE 
activity  set  typically  employed  throughout  any  system  acquisition  cycle  is  shown  in  Figure  6. 
Although  we  can  anticipate  that  many,  if  not  all,  of  the  activities  will  be  impacted  by  taking  on 
the  role  of  the  LSI,  our  interviews  have  indicated  that  the  early  application  of  discipline  SE 
practices  and  methods  create  the  greatest  and  most  significant  positive  impact  to  reducing 
risk  and  increasing  system  success.  Figure  6  shows  that  the  dark  blue  activities  are  the 
most  likely  candidates  to  receive  attention  for  workforce  development  and  to  apply  SDEA 
concepts  and  tools. 
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Figure  6.  Systems  Engineering  Activities  Need  Strengthening  to  Expand  Role  as  an  LSI 


Although  still  formative,  the  SE  activities  shown  may  focus  upon  general  concepts  of 
the  following: 

•  Concept  development — Eliminating  disconnects  between  originator  needs 
and  acquisition  system  requirements. 

•  Design — Inexperience,  insufficient  or  missing  tools,  and  weak  methods. 

•  Integrative  methods — Organizational  teaming,  SoS  awareness,  standards, 
priorities,  technical  incentives. 

•  Development — DoD  (LSI)  and  contractor  common  models  and  tools. 

•  Integration — Gaps  in  cross-discipline  skills  and  experience,  lack  of  facilities, 
weak  methods,  lack  of  jointness. 

•  Test  and  evaluation — Gaps  in  attaining  a  system  that  is  mature  and  ready  for 
test. 

System  Definition-Enabled  Acquisition  (SDEA)  System  Concept 
Top-Level  Concept 

The  top-level  SDEA  concept  is  shown  in  Figure  7.  The  intent  is  that  SDEA  supports 
all  of  the  SE  activities  in  Figure  6  in  a  quantitative  and  repeatable  manner.  The  SDEA 
system  comprises  system  definition,  modeling,  and  analyses  that  provide  repeatable  and 
quantifiable  designs.  The  SDEA  system  is  to  provide  a  data-driven  system  definition  and 
model-driven  SE  approach  that  supports  LSI  SE  and  design. 

The  SDEA  system  is  synergistic  with  the  program  definition,  system  definition, 
supportability  definition,  and  system  production.  Note  that  all  of  these  activities  support 
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baseline  development  and  control.  Program  definition  leads  to  system  definition  and  the 
handoff  contract  (documents)  associated  with  system  capabilities  and  top-level  performance 
goals. 

Additionally,  program  definition  leads  to  documentation  and  agreements  that  set  in 
motion  long-term  supportability  strategies  and  activities,  such  as  logistics,  training  and 
manpower,  and  long-term  supply  chain  strategies.  The  SDEA  system  supports  both  system 
definition  in  a  very  repeatable  and  quantifiable  manner,  as  well  as  providing  clear  detail  and 
system  reliability  and  supportability  metrics  to  the  support  system  associated  with  the 
acquisition. 


Finally,  system  production  depends  on  precise  SDEA  system  definition  in  order  to 
proceed  to  production  of  the  system  in  preparation  for  deployment. 


System 

Definition 

L 

J 

□ 


SDEA  Methods 
and  Tools 


Program 

Definition 


SDEA  (MBSE)  System 


Support 

Definition 


System 

Production 


Figure  7.  SDEA  Provides  Central  Engineering  System  Support  to  Acquisition 

(derived  from  Montogmery  et  al.,  2012) 


Summary 

Past  performance  by  contractors  performing  what  some  now  believe  are  inherently 
governmental  acquisition  engineering  functions  during  major  DoD  weapon  system 
acquisition  has  proven  problematic,  in  some  cases.  Legislation  and  policy  is  moving  DoD  to 
consider  transforming  its  engineering  role  for  major  systems  (especially  SoS)  to  that  of  the 
LSI.  The  DoD  acquisition  workforce  methods,  practices,  and  tools,  however,  need  to  be 
upgraded  and  enriched  to  achieve  this  transformation.  We  believe  that  the  integration  of 
model-based  system  engineering  (MBSE)  tools  through  an  SDEA  method  is  key  to 
supporting  the  higher  levels  of  SE  design  disciplines,  analyses,  and  baseline  control,  and 
will  contribute  to  quicker  time-to-market  and  lower  integration  and  interoperability  risks  for 
future  weapons  systems. 


In  summary: 


•  An  LSI  is  needed  where  high  system  complexity,  high  risks,  or  SoS 
integration/interoperability  are  present. 

•  DoD  acquisition  organizations  are  exploring  taking  on  more  of  the  LSI  roles. 

•  DoD  acquisition  practices  need  fortifying  to  cope  with  the  more  expansive 
levels  of  SE. 
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•  SE  methods  need  reinforcing,  and  SE  tools  (e.g.,  SDEA)  need  to  be  acquired 
and  integrated  into  the  workflow  with  capability  to  provide 

o  early  and  strong  SE  application  (pre-milestone  A), 

o  data-driven  design  support  tools, 

o  repeatable  and  quantifiable  system  design  analyses, 

o  persistent  (multi-year/multi-system)  design  data  repository, 

o  SoS  interoperability  and  integration  analyses,  and 

o  operational,  qualification  (V&V),  suitability,  and  sustainability 
design/analysis  support. 
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Abstract 

The  Deputy  Assistant  Secretary  of  the  Navy  for  Research,  Development,  Test,  and 
Evaluation  (DASN  RDT&E)  created  a  Business  Innovation  Initiative  (Bll)  to  identify  and 
overcome  challenges  presented  by  the  updated  Naval  Open  Systems  Architecture  (OSA) 
strategy.  The  Bll  seeks  to  find  innovative  ways  to  incentivize  Naval  program  management 
staff  and  their  industry  partners  to  implement  aggressive  change  measures  that  improve  cost, 
performance,  and  time  to  field  for  national  security  systems.  The  Bll  conducts  workshops  and 
crowd-sourcing  activities  that  focus  on  specific  business  impediments  to  OSA.  The  Massive 
Multiplayer  Operational  War  Game  Leveraging  the  Internet  (MMOWGLI)  game  was  used  as  a 
crowd-sourcing  tool  to  elicit  the  collective  intelligence  of  acquisition  practitioners,  students  of 
business,  and  industry  stakeholders  to  identify  and  overcome  challenges  presented  by  the 
updated  OSA  strategy.  The  MMOWGLI  platform  provides  an  efficient  mechanism  that 
crosses  functional  and  geographical  workspace  boundaries  for  exchanging  ideas  and  forming 
community  in  addressing  the  OSA  business  problem.  The  results  of  the  first  Bll  crowd¬ 
sourcing  event  using  the  MMOWGLI  platform  are  presented  in  this  report. 

Introduction 

The  Assistant  Secretary  for  Research  Development  and  Acquisition  (ASN[RDA]) 
authorized  a  new  Naval  Open  Systems  Architecture  (OSA)  strategy  in  November  2012  (see 
Appendix  A)  to  reduce  the  total  ownership  cost  of  systems,  encourage  innovation,  and  more 
rapidly  deliver  needed  capabilities  to  the  warfighter.  This  strategy  specifically  challenges  the 
Naval  acquisition  workforce  to  institute  measures  to  improve  competition,  eliminate 
redundant  developments,  and  coordinate  program  activities  that  promote  the  reuse  of 
tactical  products  across  sea  and  air  platforms.  The  acquisition  organization  is  tasked  to 
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implement  the  strategy,  and  success  will  require  substantial  changes  in  the  Navy’s  business 
practices,  organizational  structures,  and  resource  planning. 

In  concert  with  the  updated  strategy,  the  Deputy  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  Test,  and  Evaluation  (DASN  RDT&E)  created  a  Bll  to  search  for 
ways  to  better  incentivize  OSA  business  practices  across  our  current  programs  of  record. 
Mr.  Sean  Stackley,  ASN(RDA),  said  in  a  recent  article, 


The  value  of  an  innovation  initiative  is  to  explore  what  business  relationship 
changes  are  needed  to  open  up  competition;  incentivize  better  contractor 
performance;  increase  access  to  innovative  products  and  services  from  a 
wider  array  of  sources;  decrease  time  to  field  new  capabilities;  and  achieve 
lower  acquisition  and  life-cycle  costs  while  sustaining  fair  industry  profitability. 
(Hudson,  2012) 


The  Updated  Naval  OSA  Strategy  and  the  Bll 
Relationship 
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Figure  1.  Updated  Naval  Open  Systems  Architecture  Strategy  and 
Business  Innovation  Initiative  Relationship 


As  a  companion  of  the  updated  Naval  OSA  strategy,  the  Bll  examines  business 
operations,  processes,  and  incentives  associated  with  delivering  national  security  systems. 
DASN  RDT&E  recognized  that  using  the  crowd-sourcing  war  game  facility  developed  by  the 
Naval  Postgraduate  School  (NPS)  presented  an  opportunity  to  examine  both  the  OSA 
strategy  and  the  Bll  in  tandem.  Both  share  activities  that  could  be  explored  and  melded 
together  into  a  comprehensive  set  of  recommendations  on  how  to  advance  technical  and 
business  changes  needed  in  Naval  acquisition. 
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The  Bll  addresses  a  two-year  plan  that  will  codify  methods  to  rapidly  bring  innovative 
capabilities  to  the  warfighter  and  find  a  process  that  seeks  continuous  cost  improvements 
for  initial  acquisition,  production,  and  sustainment,  while  fostering  innovation  from  a  broad 
range  of  sources. 

In  the  first  year,  the  Naval  OSA  Enterprise  Team  (ET)  is  promoting  business  models 
that  lower  overall  cost  and  ensure  a  reasonable  profit  potential,  reduce  time  to  market,  and 
remove  competitive  restrictions.  This  innovation  initiative  is  looking  for  reforms  that  create 
effective  business  relationships  between  government  and  industry  (government-to-business) 
and  between  government  organizations  (government-to-government). 


Naval  OSA  Priorities: 

Rapid  Capabilities  to  the  Warfighter. 
Continuous  Cost  Improvement. 

Increased  Innovation  from  a  Broad  Range  of 
Sources 


Promote  Business  Models  That: 

--  Increases  Profit  Potential 
-Reduce  Time  to  Market 
-Remove  Competitive  Restrictions 


Government  to  Business  Reform 
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Figure  2.  Naval  Open  Systems  Architecture  Business  Innovation  Intiative 
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Government  to  Business:  In  the  first  year,  the  government-to-business  team  is 
acting  to  create  an  open  business  model  framework  that  recognizes  that  the  defense  market 
is  changing  and  seeks  to  find  opportunities  to  improve  and  normalize  business  relationships. 
Team  1  consists  of  academia  (top-tier  business  schools  and  NPS),  industry  representatives 
(strategically  formed  to  have  a  balanced  representation  with  participation  from  large  and 
small  businesses),  and  key  Department  of  the  Navy  (DoN)  acquisition  staff.  The  team  is 
scheduled  to  complete  its  objectives  within  12  months  and  publish  its  findings  and 
recommendations  to  the  ASN(RDA). 

Government  to  Government:  The  second  team  will  complement  the  work  of  the 
first  team  and  will  focus  on  changes  that  affect  the  government  acquisition  community. 

Team  2  will  consist  of  key  representatives  from  the  resource  community,  the  fleet,  and  the 
acquisition  force  (PEOs,  SYSCOMs,  and  laboratories).  The  team  will  deliver  actionable 
supporting  policies,  new  procedures,  workforce  education  /outreach  tools,  and  candidate 
legislative  changes. 

Bll  War  Games 

The  term  “war  game”  is  used  to  describe  the  crowd -sourcing  concept  exploration 
events  of  the  Bll.  The  war  game  is  a  social  networking  construct  to  explore  and  solve 
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“wicked  problems”  (Conklin,  2005)  in  a  large  and  diverse  forum.  Wicked  problems  are  those 
with  complex  interdependencies  where  the  effort  to  solve  the  stated  problem  may  reveal  or 
create  other  problems.  In  Naval  OSA,  for  example,  a  recognized  problem  is  “vendor  lock.” 
Vendor  lock  is  the  business  principle  associated  with  limited  acquisition  choice  and  sole- 
source  contracting  and  may  further  be  an  indication  of  an  unrecognized  strategic  problem. 

The  DASN  RDT&E  selected  the  Massive  Multiplayer  Operational  War  Game 
Leveraging  the  Internet  (MMOWGLI)  as  the  mechanism  to  bring  innovative  concepts  from  a 
diverse  group  together  rapidly  and  effectively.  MMOWGLI  is  an  open-source  software 
platform  sponsored  by  the  Office  of  Naval  Research  (ONR),  developed  by  NPS,  and 
originally  designed  by  Institute  For  The  Future  (IFTF).  MMOWGLI  provides  a  text-based 
social  networking  platform  that  allows  many  users  to  interact  directly  with  one  another  using 
web  browsers  in  real  time. 

The  first  Bll  MMOWGLI  game  was  intended  to  serve  as  a  trial  run  for  understanding 
whether  a  crowd-sourcing  method  for  exploring  innovations  for  implementation  of  the  Naval 
OSA  strategy  might  be  valuable  and  robust.  Round  One  of  the  Bll  MMOWGLI  game  is 
described  in  detail  in  Section  2. 
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Figure  3.  Relative  Timeline  for  Open  Systems  Architecture/Business  Innovation  Initiative 

Interactivities 


Bll  MMOWGLI  Game 
What  Is  MMOWGLI? 

MMOWGLI  is  an  online  game  platform  designed  to  elicit  collective  intelligence  from 
an  engaged  pool  of  players  to  solve  real-world  problems.  MMOWGLI  was  developed  at  the 
NPS’s  MOVES  Institute  under  the  sponsorship  of  the  ONR.  It  was  launched  in  201 1 . 
MMOWGLI  games  have  been  conducted  to  look  for  innovative  ideas  in  a  variety  of  complex 
problems,  including  countering  international  maritime  piracy,  improving  energy  efficiency  on 
Navy  ships,  and  several  others.  Games  have  also  been  conducted  at  a  variety  of  scales  (50 
to  550  players)  and  with  different  audiences  (public,  industry,  Navy,  and  others). 
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Bll  MMOWGLI  Round  One 

War-game  design  is  an  important  part  of  the  process.  The  MMOWGLI  platform  is 
designed  to  spur  innovation  and  is  highly  configurable,  but  players  will  not  visit  or  engage 
unless  a  clear  purpose  is  evident.  Several  months  of  effort  went  into  defining  and  refining 
the  game  themes  and  prospective  audience  that  might  provide  the  greatest  possible  impact 
supporting  the  overall  project  goals.  A  preliminary  workshop  and  a  trial  mini-game  provided 
excellent  feedback  regarding  what  themes  were  most  interesting.  Extensive  details  and  the 
game  itself  are  available  online  (see  Bll  MMOWGLI  Game  Portal,  n.d.;  Bll  MMOWGLI  Game 
Blog,  n.d.;  Bll  MMOWGLI  Game  Play,  n.d.;  Bll  MMOWGLI  Game  Reports,  n.d.). 

The  DASN  RDT&E  conducted  the  first  of  two  planned  Bll  MMOWGLI  games  for 
fiscal  year  (FY)  2013  during  the  period  of  January  14  through  28.  The  purpose  of  the  first 
round  was  to  test  and  validate  the  use  of  the  MMOWGLI  crowd-sourcing  tool  for  finding 
innovation  in  Naval  acquisition  and  to  identify  how  to  use  it  to  find  challenges  of 
implementing  the  updated  Naval  OSA  strategy.  Over  100  professionals  from  government, 
academia,  and  industry  participated,  generating  over  890  idea  cards  and  1 1  action  plans. 
The  first  week  was  dedicated  to  card  play,  followed  by  a  second  week  used  for  detailed 
action-plan  development. 

The  Call  to  Action  Statement 

Establishing  group  motivation  and  common  goals  is  a  fundamental  prerequisite  for 
an  effective  war  game.  The  Bll  MMOWGLI  game  was  introduced  to  players  with  this 
invitation: 


The  BII  MMOWGLI  game  is  for  professionals  exploring  how  best  to  achieve  the 
business  goals  of  the  Navy’s  new  “Open  Systems  Architecture  (OSA)  Strategy.”  Your 
feedback  on  this  plan  is  welcome. 

Round  One  explores  the  challenges  and  opportunities  in  developing  a  “Payloads  over 
Platforms”  business  model  driven  by  the  OSA  strategy.  We  are  here  to  discover  ways  to 
incentivize  acquisition  programs  and  our  industry  partners  to  help  craft  a  new  business 
relationship  that  eliminates  redundant  development,  ensures  sustainability  of  war-fighting 
capabilities,  and  positively  rewards  good  industry  perfonnance. 

Your  contributions  are  essential.  Please  join  in.  The  BII  portal  is  a  great  information 
resource  for  game  play.  Check  the  BII  blog  for  game  news.  Thanks  for  your  ideas. 

Play  the  game,  change  the  game. 


Card  Play 

All  MMOWGLI  games  begin  with  a  set  of  thought-provoking  seed  cards  to  initiate 
discussion.  In  the  Bll  game  Round  One,  the  seed  cards  were  labeled  as  “challenges.”  Each 
challenge  card  was  configured  to  provide  four  responses:  expand,  counter,  adapt,  or 
explore.  Players  would  select  a  response  category  and  type  their  thoughts  in  140  characters 
maximum,  which  forced  them  to  be  clear  and  concise  with  their  message.  Other  players 
could  immediately  see  that  card  as  part  of  the  discussion  chain  and  be  able  to  select  it  and 
respond  in  kind:  expand,  counter,  adapt,  or  explore.  These  linked  cards  created  a  chain  (or 
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discussion)  where  any  number  of  players  could  contribute.  Rules  on  how  to  play  a  particular 
response  card  were  left  to  the  player’s  intuition.  See  Figure  4. 
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Figure  4.  Bll  MMOWGLI  Card  Chain  Exemplar 


Challenge  Cards.  In  our  Bll  MMOWGLI  game,  seven  general  questions  about  the 
strategy  were  cast  as  challenge  cards,  intended  to  help  players  begin  conversations  of 
interest.  They  were  as  follows: 


Challenge  Cards: 

1 .  What  are  the  primary  challenges  of  the  CNO’s  “payloads  over  platforms” 
strategy? 

2.  How  to  incentivize  programs  and  industry  to  deliver  reusable  component 
solutions  as  an  acquisition  model? 

3.  How  can  we  align  across  programs  to  eliminate  redundant  development? 

4.  How  can  Naval  acquisition  align  programs  to  a  limited  number  of  technical 
reference  frameworks  (TRFs)? 

5.  Technology  conditions  change  over  long  service  lives  of  ships  and  aircraft. 

6.  Has  open  architecture  (OA)  succeeded  from  a  micro  perspective  and  failed  in 
the  macro  perspective? 

7.  What  do  you  see  as  the  greatest  challenge  moving  into  Phase  1  of  the  OSA 
strategy? 

An  additional  set  of  seven  seed  cards  were  also  formed  under  the  heading  Future 
Goals.  They  were  “what  do  you  think?”  topics  intended  to  stimulate  discussion  on  system 
acquisition.  They  were  as  follows: 
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Future  Goals  Cards: 


1.  How  might  the  acquisition  process  change  to  enable  more  competition  by 
industry:  large  companies,  small  companies,  and  system  integrators? 

2.  How  can  life  cycle  management  (LCM)  improve  to  avoid  “lock  in”  and  enable 
competition  through  the  long  term  program? 

3.  What  ships  and  aircraft  are  the  best  exemplar  platforms  to  consider? 

4.  What  Navy  programs  are  the  best  exemplar  payloads  to  consider? 

5.  Unmanned  systems  have  much  shorter  lifecycles,  enabling  rapid  change. 
How  might  that  improve  the  acquisition  process? 

6.  How  do  we  define  payloads?  One  person’s  payload  is  another  person’s 
truck... 

7.  What  topic  do  you  want  to  see  addressed  in  round  2  of  the  Game  next 
summer?  Our  theme:  Future  Paths  Forward  For  Improved  Business 
Practices. 


Game  Play  Results 

Figures  5  and  6  show  the  number  of  player  responses  for  each  of  the  seed  cards. 
Expand  and  explore  cards  were  most  often  chosen  by  the  players  to  amplify  on  a  previous 
card.  Counter  and  adapt  card  responses  were  given  much  less  frequently. 


Challenge  Card  Chains 


3  4 

Seed  Card 


i  Expand 
i  Counter 
Adapt 
l  Explore 


Figure  5.  Challenge  Card  Chains 
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Future  Goals  Card  Chains 


Figure  6.  Future  Goals  Card  Chain 

Four  seed  cards  generated  the  most  response.  It  is  noted  that  these  cards  focused 
more  on  Phases  2  and  3  of  the  OSA  strategy. 

Challenge  Seed  Card  #2:  How  to  incentivize  programs  and  industry  to  deliver 
reusable  component  solutions  as  an  acquisition  model? 

Finding  ways  to  incentivize  programs  and  industry  to  deliver  reusable  component 
solutions  as  an  acquisition  model  generated  the  second  highest  interest  in  the  game  with 
151  total  exploration  cards  played.  Today,  acquisition  programs  most  often  employ  effort- 
based  development  contracts  for  a  single  end  user  without  considering  how  it  might  be 
reused  in  other  programs.  A  new  concept  of  a  “first  user-subsequent  user”  model  was 
offered  as  a  means  to  include  the  subsequent  user  in  the  acquisition  model  where  vendor 
savings  naturally  occur,  indicating  how  the  Navy  and  industry  might  share  benefits  of  reuse. 
Conversations  focused  on  data  rights,  intellectual  property,  licensing,  royalties,  and  RFP 
process  areas.  Industry  has  the  incentive  of  an  expanded  market,  and  government  has  the 
incentive  of  reduced  schedule  and  cost  profiles. 

Challenge  Seed  Card  #3:  How  can  we  align  across  programs  to  eliminate 
redundant  development? 

This  topic  drew  the  highest  response  in  the  game  with  157  exploration  cards  played. 
Program  objectives  memoranda  (POM)  roadmaps  are  a  resource  management  tool  that 
align  program  funding  to  the  corresponding  Future  Years  Defense  Program  (FYDP). 
Financial  plans  for  product  line  acquisitions  were  offered  as  a  way  to  show  sponsors  that 
collaboration  equals  value.  These  would  include  first  user-future  user  program 
budget/schedules  as  a  way  to  help  PEOs  and  resource  sponsors  connect  the  dots. 

Challenge  Seed  Card  #7:  What  do  you  see  as  the  greatest  challenge  in  executing 
Phase  1  of  the  OSA  strategy? 
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One  player  cited  Dr.  Gansler’s  “The  Changing  Face  of  Competition,”  noting  that  one 
third  of  the  acquisition  workforce  has  less  than  five  years  of  experience  and  does  not 
understand  industry  operations  and  incentives.  As  a  result,  government  relies  on  a 
contracting  style  based  more  on  precedence  (i.e. ,  how  we  do  things  today  rather  than  how 
we  might  do  things  better).  Players  focused  on  the  need  for  specific  workforce  training  for 
OSA  as  the  greatest  challenge  to  the  strategy. 

Challenge  Seed  Card  #4:  How  can  Naval  acquisition  align  programs  to  a  limited 
number  of  technical  reference  frameworks  (TRFs)? 

Citing  declining  budgets  as  a  motivator,  it  was  noted  that  some  PEOs  are  starting  to 
coordinate  technical  frameworks  into  their  plans  and  programs.  Even  though  PEOs 
collaborate,  there  is  no  technical  chain  of  command,  so  aligning  frameworks  becomes  a 
coalition  of  the  willing.  Players  suggested  having  the  government  own  their  TRFs,  so 
product  developers  have  a  stable  base  to  enable  broader  competition.  A  counterargument 
stated  that  “a  limited  number  should  not  be  the  goal  since  different  programs  have  different 
needs.”  This  was  quickly  challenged  by  another  player:  “A  different  need  is  code  for  not 
invented  here.”  The  message  from  this  discussion  thread  is  that  coordinated  TRF  design 
and  governance  will  form  a  key  role  in  the  OSA  strategy. 

Action  Plan  Products 

Idea  card  chains  quickly  illustrate  the  pros,  cons,  and  alternatives  associated  with  an 
issue.  More  is  needed  when  moving  beyond  improved  understanding  towards  charting 
possible  paths  forward  and  analyzing  alternative  courses  of  action. 

In  MMOWGLI,  an  action  plan  is  an  expansion  of  an  idea  formed  in  game  play.  It  is 
authored  by  a  small  group  of  players,  usually  three  or  four,  who  collaborate  to  describe  how 
an  idea  might  work,  what  the  benefits  are,  and  the  scope  of  effort  necessary.  There  are  five 
parts  to  an  action  plan  (essentially,  who,  what,  where,  when,  why  and  how).  Below,  the  “how 
will  it  work”  section  is  included  as  a  summary  of  the  plan. 

The  following  five  action  plans  were  significant  in  that  they  achieved  a  greater  than 
66%  (two  of  three  possible  thumbs-up)  voter  rating.  Players  that  vote  one  thumb-up  think 
that  the  plan  is  not  ready  to  proceed.  Players  that  give  two  thumbs-up  consider  an  action 
plan  worthwhile,  while  players  awarding  three  thumbs-up  mean  that  they  think  that  a  plan  is 
especially  promising  and  worthy  of  further  exploration.  Here  are  the  top  five  plans,  based  on 
averaged  player  ratings,  from  highest  to  lowest: 

Action  Plan  #10:  How  can  the  Navy  use  data  and  software  rights  to  achieve  the 
OSA  strategy,  ensuring  long-term  stability  and  growth? 

How  will  it  work?:  Conduct  a  series  of  workshops  and  discussion  groups  to  focus  on 
different  perspectives,  including  how  we  best  leverage  the  commercial  market  (while 
addressing  issues  and  challenges  unique  to  DoD  systems)  and  how  we  establish  a 
marketplace  for  reusable  applications  to  reduce  cost  and  increase  innovation  and 
interoperability.  Result  is  a  spectrum  of  strategies  from  which  programs  can  select 
depending  on  the  reuse  potential  of  specific  products  in  identified  markets.  Then  train  the 
workforce. 

Action  Plan  #3:  Define  community  of  interest  (CO!)  activities  and  external 
relationships  in  the  context  of  the  OSA  strategy. 

How  will  it  work?:  The  main  catalyst  for  COIs  today  is  interoperability  for  mission 
performance,  rather  than  reuse.  The  mission  area  interoperability  and  integration  (l&l)  effort 
should  help  define  systems  of  systems  (SoS)  capability  gaps.  Once  people  start  thinking  in 
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SoS  terms,  they  can  start  looking  at  reuse  across  platforms.  That  often  leads  to  a 
homegrown  approach  and  a  PARM  relationship  between  programs.  PEOs  submit  capability- 
based  coordinated  POM  inputs  for  whole  capabilities.  The  OPNAV  coordinates  funding 
decisions  so  that  holistic  upgrades  are  funded. 

Action  Plan  #11:  Streamline  OSA  contracting  to  make  the  process  more  agile, 
rewarding  innovation. 

How  will  it  work?:  Unlike  the  SBIR  program,  all  efforts  would  be  fully  funded,  but  they 
would  be  similar  to  SBIRs  in  that  they  are  incrementally  funded.  Initial  awards  based  on  start 
year  (e.g.,  2014  base  year,  with  option  years  1  ...  5).  Base-year  candidates  would  be 
minimally  funded  with  data  rights  and  royalties  for  reuse,  while  more  mature  option  year 
awards  would  benefit  from  increased  funding.  This  would  push  competition  for  best  solutions 
to  win  the  options  as  well  as  the  additional  royalties. 

Action  Plan  #8:  Outline  ROI  steps  to  justify  reusing  components  across  participating 
COI  programs. 

How  will  it  work?:  An  ROI  metric  would  quantify  the  different  ways  we  save  via  reuse, 
including  requirements  specification,  integration,  test,  training,  maintenance,  sparing  (for 
H/W),  IAVM,  and  technology  refresh.  COPLIMO  has  a  COCOMO-based  cost  model  that  can 
look  at  SW  product  lines  and  show  where  costs  are  reduced  with  subsequent  reuse  by 
leveraging  existing  requirements.  Also,  we  need  a  way  to  quantify  operational  benefits  from 
consistency  of  performance,  user  interfaces,  and  more  widespread  fielding. 

Action  Plan  #6:  How  do  programs  that  group  together  get  rewarded  for  increasing 
enterprise  value? 

How  will  it  work?:  In  this  budget  climate,  PMs  will  seek  protection,  even  at  the  cost  of 
increased  program  risk,  by  coupling  budget  and  schedules  together.  Top  leadership 
direction  will  be  needed  to  ensure  that  the  reward  mechanisms  are  impactful  and 
consistently  applied.  Enterprise  value  must  be  held  to  a  very  high  and  consistent  standard. 
Accolades  must  be  peer-reviewed.  This  would  have  two  benefits:  the  attention  to  award 
criteria  will  rise,  and  consistency  of  awards  will  increase. 

Lexical  Link  Analysis 

Results  were  analyzed  independently  at  NPS  using  the  lexical  link  analysis  (LLA) 
software-based  text  comparison  tool.  LLA  compared  Bll  game  data  to  the  OSA  strategy 
document  and  separately  to  the  Department  of  Defense  (DoD)  Open  System  Architecture 
Contract  Guidebook  for  Program  Managers  (Defense  Acquisition  University  [DAU],  n.d.). 

LLA  metrics  indicate  that  the  OSA  strategy  is  not  considered  risky  and  not  particularly 
controversial,  nor  is  it  considered  impossible  to  implement. 
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What  does  LLA  indicate  about  bii  MMOWGLI  Round  1? 

•  LLA  analysis  of  MMOWGLI  data  indicates 

—  Ideas  and  draft  Action  Plans  expressed  in  bii  game,  by 
anonymous  players,  showed  strong  consistency  with  the 
concepts  in  Program  Manager's  Contract  Guidebook 

—  Metrics  indicate  draft  OSA  strategy  triggered  new  and 
innovative  ideas 

—  Metrics  did  not  indicate  that  OSA  strategy  was  risky, 
controversial,  or  impossible  to  implement  etc. 

•  Next-round  game  efforts: 

—  Focus  on  LLA-discovered  themes  of  particular  interest 

—  Encourage  even  more  player  input 


Figure  7.  LLA  Conclusions,  BII  MMOWGLI  Analysis  Report,  February  6,  2013 

LLA  is  a  form  of  text-based  data  mining  in  which  word  meanings  represented  in 
coupled  lexical  terms  (e.g.,  word  pairs)  can  be  represented  as  if  they  are  in  a  community  of 
a  word  network.  LLA  discovers  and  displays  these  networks  of  word  pairs  from  large-scale, 
unstructured  data.  LLA  can  also  be  used  as  a  search  and  knowledge  management  tool  for 
scoring  and  ranking  interesting  information  and  for  visualizing  and  reporting  correlations 
among  categories  and  layers  of  information  including  lexical,  semantic,  and  social  links. 

This  effort  then  presents  the  decision-maker  with  results  that  were  previously 
unavailable  regarding  emerging  patterns  and  themes,  as  well  as  unprecedented  levels  of 
analysis,  thus  reducing  the  workload  and  overcoming  the  blind  spots  of  human  analysts  by 
leveraging  automation.  For  example,  for  the  recent  MMOWGLI  games  used  to  develop  and 
identify  new  ideas  about  a  specific  subject,  LLA  was  engaged  to  identify  potentially 
interesting  information  from  idea  cards,  link  them,  and  then  reveal  them  to  domain  experts. 
The  methodology  of  LLA  is  discussed  in  more  detail  in  the  article  in  these  same  conference 
proceedings  (Zhao,  Brutzman,  &  MacKinnon,  2013). 

For  the  BII  MMOWGLI  project,  we  performed  two  separate  post-game  data  analyses. 

1 .  Idea  cards  (892)  and  action  plans  (11)  were  compared  to  the  proposed  OSA 
strategy  (four  pages)  considered  by  players. 

2.  Idea  cards  (892)  and  action  plans  (1 1 )  were  compared  to  the  Department  of 
Defense  (DoD)  Open  System  Architecture  Contract  Guidebook  for  Program 
Managers  (158  pages)  already  familiar  to  most  players. 

What  did  LLA  indicate  about  BII  MMOWGLI  Round  One?  LLA  data  analysis  indicates 
the  following: 
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•  Ideas  and  draft  action  plans  expressed  in  the  Bll  game  by  anonymous 
players  showed  strong  consistency  with  the  concepts  in  Department  of 
Defense  (DoD)  Open  System  Architecture  Contract  Guidebook  for  Program 
Managers. 

•  Metrics  indicate  that  draft  OSA  strategy  triggered  new  and  innovative  ideas. 

•  Metrics  did  not  indicate  that  OSA  strategy  was  risky,  controversial,  nor 
impossible  to  implement. 

LLA  also  discovered  eight  main  or  popular  themes  listed,  reflecting  common  interest 
of  the  players,  using  the  following  keywords: 

•  multiple  support  and  components 

•  common  data,  data  model 

•  component  reuse,  OSA 

•  open  system  and  business 

•  systems  architecture,  current  systems 

•  specific  price  and  fee 

•  existing  reusable  programs 

•  engineering,  government,  and  community 

We  also  found  innovative  ideas  (i.e. ,  gaps  between  the  game  data  and  the  OSA 
strategy  document)  in  the  following  areas  (themes): 

•  small  and  shared  based 

•  developed  and  built  faster 

•  critical  definition 

•  specific  price  and  fee 

•  sponsors  change  and  risk 

•  changing  requirements 

•  interoperability  and  interfaces 

Figure  8  shows  one  example  theme  detailed  from  the  comparison  of  game  data  with 
the  OSA  strategy  document.  Red  nodes  show  the  top  three  word  hubs  with  most  links  (most 
“central”).  Yellow  word  pairs  are  unique  to  the  action  plans,  green  word  pairs  are  unique  to 
the  ideas  cards,  and  blue  word  pairs  are  unique  to  the  OSA  strategy  document.  Red  word 
pairs  are  found  in  more  than  two  sources. 
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Figure  8.  One  Theme  Detailed  From  the  Comparison  of  the  Game  Data  With  the  OSA 

Strategy  Document 

Note.  Red  nodes  show  the  top  three  word  hubs  with  most  links  (most  “central”).  Yellow  word  pairs  are 
unique  to  the  action  plans,  green  word  pairs  are  unique  to  the  ideas  cards,  and  blue  word  pairs  are 
unique  to  the  OSA  strategy  document.  Red  word  pairs  are  found  in  more  than  two  sources. 

Subject-matter  expertise  is  necessary  to  determine  whether  statistically  generated 
concepts  have  real  relevance  and  interest.  Based  on  the  concepts,  themes,  and  gaps 
discovered  in  Bll  Round  One,  analysts  compiled  and  distilled  a  list  of  20  candidate  topics  for 
possible  seed  card  exploration  in  Bll  Round  Two.  One  example  is  shown  as  follows,  using 
the  theme  illustrated  in  the  preceding  figure  and  centered  around  the  keywords  multiple- 
support  and  components. 

•  How  will  the  concept  of  TRF  addressed  in  the  strategy  document  be 
enhanced  from  the  ideas  surrounding  “multiple  support  and  components,”  a 
COTS-based  model? 

•  Relevant  concepts  from  Round  One:  Network-based  COTS  (Idea  Card  Chain 
850),  COTS  hardware  (Idea  Card  Chains  138,  89),  COTS  insertion  (Idea 


m  khsiikm 
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Card  Chain  770),  COTS  model  (Action  Plan  4),  and  COTS  elements  (Action 
Plan  10) 

The  reason  emerging  concepts  such  as  this  can  be  generated  is  that  salient  ideas 
emerge  from  commonality  and  differences  between  data  corpus  comparisons,  allowing 
expert  analysts  to  discern  which  topics  might  hold  the  greatest  interest. 

A  number  of  reports  and  analytic  products  are  generated  by  the  game.  To  reap  the 
maximum  benefits  from  the  contribution  of  so  many  focused  ideas  and  plans,  post-game 
analysis  is  an  important  activity  that  helps  find  the  most  valuable  conclusions  and  also 
results  in  products  of  greatest  value  to  past  (and  future)  participants. 

Lessons  Learned 

Round  One  Results  Exceeded  Expectations.  The  DASN  RDT&E  expected  that  Bll 
MMOWGLI  Game  Round  One  would  generate  a  list  of  challenges  for  the  updated  OSA 
strategy.  Players  instead  concentrated  their  activities  on  identifying  ways  to  make  the 
strategy  successful.  It  was  clear  that  they  viewed  their  role  as  problem  solvers  rather  than 
problem  identifiers.  This  is  a  valuable  lesson  for  planning  Round  Two  of  the  Bll  MMOWGLI 
game.  Understanding  a  player’s  natural  motivation  to  create  ideas  on  challenges  gives  the 
planning  team  a  better  understanding  of  how  to  select  and  “seed”  the  topics  for 
consideration  in  the  next  round  of  play. 

Crowd  Sourcing  Builds  Community.  An  unexpected  benefit  from  using  this  tool 
was  the  aspect  of  community  building.  Those  who  played  Bll  MMOWGLI  Game  Round  One 
formed  a  diverse  team  with  different  points  of  view.  Game  players  were  anonymous  and 
used  a  made-up  player  name  to  engage  in  discussions.  The  anonymity  provided  a  shield 
from  intimidation  or  prejudice,  in  that  all  seemed  open  to  the  ideas  and  comments  of  others 
and  willing  to  engage  in  debate  on  the  pure  basis  of  the  issues,  rather  than  taking  an 
industry  or  government  positional  argument.  Players  expanded  their  ideas  using  a  game 
feature  called  action  plans,  inviting  other  players  to  help  them  author  potential  solutions  ... 
and  it  worked.  The  lesson  learned  here  is  to  open  the  problem  set  to  the  widest  audience 
possible.  Opinions  and  interactions  ultimately  come  together  to  build  the  views  of  a  larger 
community. 

Enlisting  a  Broad  Audience  Requires  Promotion  and  Advertising.  Bll 

MMOWGLI  Game  Round  One  was  as  much  about  testing  the  tool  as  it  was  about  getting 
direct,  actionable  results.  As  such,  marketing  was  minimal.  E-mail  invitations  shortly  ahead 
of  the  event  and  a  few  public  announcements  formed  the  ad  campaign.  Current  members  of 
the  OSA  ET,  Bll  academic  partners,  and  selected  industry  partners  who  volunteered  to 
participate  at  the  Defense  Daily  Open  Architecture  Summit  of  October  2012  formed  the  main 
body  of  invitees.  In  response  to  approximately  900  e-mail  invitations,  just  over  100  people 
played. 

Conclusions 

The  success  of  the  new  Naval  OSA  strategy  relies  on  the  Navy  acquisition 
community’s  ability  to  cross  program  boundaries  and  work  together  toward  common  goals. 
We  can  take  full  advantage  of  the  power  of  community  by  using  focused  information 
technology.  MMOWGLI  has  proved  to  be  an  effective  mechanism  to  improve  communication 
and  coordination  among  the  various  program  offices,  program  executive  offices,  and 
systems  commands.  The  DASN  RDT&E  considers  the  Bll  MMOWGLI  Game  Round  One  a 
success  for  the  following  reasons: 
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1 .  The  use  of  MMOWGLI  to  explore  Naval  OSA  challenges  exceeded 
expectations.  The  players  of  the  Bll  war  game  identified  several  innovations 
on  how  to  implement  the  Naval  OSA. 

2.  Game  play  supported  the  intent  of  the  Bll  to  explore  the  business 
relationships  changes  to  (a)  open  up  competition,  (b)  incentivize  better 
contractor  performance,  (c)  increase  access  to  innovative  products  and 
services  from  a  wider  array  of  sources,  (d)  decrease  time  to  field  new 
capabilities,  and  (e)  achieve  lower  acquisition  and  life-cycle  costs  while 
sustaining  fair  industry  profitability. 

3.  Those  who  played  Bll  MMOWGLI  Game  Round  One  formed  an  enthusiastic 
group  with  different  points  of  view,  highlighting  the  power  of  crowd  sourcing  to 
build  a  diverse  community  around  topics  of  mutual  interest. 

Multiple  recommendations  for  future  work  are  now  being  pursued  as  the  team 
prepares  for  Bll  MMOWGLI  Round  Two.  Our  motto  remains:  Play  the  game,  change  the 
game! 
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Appendix  A 


Introduction 

The  current  Naval  Enterprise  acquisition  model  is  centered  on  highly  integrated  platforms  with 
systems  that  are  largely  vendor  locked,  and  expensive  to  acquire  and  upgrade.  This  model  is  especially 
problematic  in  the  current  economic  environment. 

The  Naval  Open  Systems  Architecture  (OSA)  strategy  will  decompose  monolithic  business  and 
technical  designs  into  manageable  product  lines  composed  of  competition-driven  modular  Enterprise 
components.  This  will  yield  innovation,  reduced  cycle  time,  and  lower  total  ownership  costs. 

The  New  Naval  Enterprise  OSA  Strategy: 

The  Naval  OSA  Strategy  is  an  iterative  set  of  business  and  technical  changes  that  points  to 
an  end  state  where  affordable,  open  platforms  easily  accommodate  open  modules.  As  the  Navy 
moves  toward  this  future,  the  Enterprise  must  first  align  itself  to  become  open,  modular,  common, 
competitive,  and  ultimately,  affordable.  It  will  begin  by  implementing  change  in  a  coordinated 
fashion  across  all  programs. 

The  Naval  OSA  Enterprise  Team  will  lead  the  execution  of  this  strategy  with  the  participation  of 
stakeholders  (e.g..  Resource  Sponsors,  PEOs,  TWAs,  etc.)  as  follows: 

•  Implement  the  coordinated  set  of  business  changes  that  improve  competition,  incentivize 
better  performance,  and  deliver  capability  more  rapidly; 

•  Construct  a  limited  number  of  technical  reference  frameworks  to  immediately  support 
improved  competition  and  ultimately  enable  enterprise  re-use; 

•  Develop  an  Execution  Guidebook  for  this  strategy;  and 

•  Lead  and  guide  training  the  workforce  on  OSA  implementation. 

Once  these  changes  have  been  adopted  at  the  program  level,  a  second  iteration  (Figure  1)  will  prepare 
the  Enterprise  to  eliminate  redundancy  and  deliver  open  systems  with  reusable  modules. 


Implementing  The  CNO's  Vision  of  “Trucks  and  Payloads' 


Figure  1.  Iterative  Naval  OSA  Strategy 


Open  Systems  Architecture  (OSA)  Strategy 
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ASN-RD&A 


An  Open  Systems  Architecture  (OSA)  approach  integrates  business  and  technical  practices  that 
yield  systems  with  severable  and  compete-able  modules.  A  system  constructed  in  this  way  allows 
vendor- independent  acquisition  of  warfighting  capabilities,  including  the  intentional  creation 
of  inter- operable  Enterprise- wide  reusable  components.  Successful  OSA  acquisitions  result  in 
reduced  total  ownership  costand  can  be  quickly  customized,  modified,  and  extended  throughout 
the  product  life  cycle  in  response  to  changing  user  requirements. 

Naval  OSA  Strategy  Actions 

Unless  otherwise  stated,  the  principal  lead  for  the  following  actions  is  the  Open  Systems 
Architecture  Enterprise  Team  (OSAET),  led  by  Deputy  Assistant  Secretary  of  the  Navy  (Research, 
Development,  Test  and  Engineering)  (DASN  RDT&E),  with  participation  from  Systems  Commands 
(NAVSEA,  NAVAIR,  SPAWAR,  and  MARCORSYSCOM),  program  Executive  Offices 
(PEO),  the  Office  of  Naval  Research,  and  designated  Communities  of  Interest  (COIs). 


Phase  1:  Align  current  programs  to  execute  the  OSA  strategy  and  report  progress. 


Phase  2:  Consolidate  technical  frameworks  across  programs ;  eliminate  redundant  stovepipes. 
Phase  3:  Implement  Enterprise  architecture  of  modular  development  and  maximum  reuse. 

Phase  1  Tasks 

•  Establish  a  limited  number  of  OSA  Technical  Reference  Frameworks  (TRFs); 

•  Change  acquisition  processes  to  adopt  these  coordinated  and  common  OSA  TRFs; 

•  Require  and  incentivize  competition  throughout  program  life  cycles; 

•  Establish  meaningful  metrics  to  assess  OSA  progress; 

•  Develop  the  OSA  Strategy  Execution  Guidebook; 

•  Educate  the  Naval  Engineering,  Logistics,  and  Acquisition  Workforce;  and 

•  Codify  knowledge,  skills  and  processes  in  the  “OSA  Program  Managers  Guidebook,  rev  1.0” 

Phase  2  Tasks 

Tasks  for  phase  2  are  TBD ;  here  are  a  few  categories: 

•  Communications  processes  to  provide  transparency  across  PEOs  and  SYSCOMS  about 
existing  programs 

•  Incentives  for  collaboration  and  cooperation 

•  Funding  techniques  for  cross- Enterprise  co-development 

•  Update  the  “OSA  Program  Managers  Guidebook” 

•  Build  on  education  efforts  through  DAU  and  integrate  the  new  Guidebook  with 
standing  courses 

Phase  3  Tasks 

Tasks  for  phase  3  are  TBD;  here  are  categories: 

•  Fine  tune  communications  processes  to  provide  transparency  across  PEOs  and  SYSCOMS 
about  existing  programs,  as  needed 

•  Adjust  incentives  for  collaboration  and  cooperation  as  needed 

•  Add  courses  to  fill  needed  knowledge  gaps 

•  Adjust  funding  techniques  for  cross- Enterprise  co-development 


Assistant  Secretary  of  the  Nary  for  Research ,  Development  and  Acquisition 
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Establish  a  Limited  Number  of  Technical  Reference  Frameworks  (TRFs) 

A  Technical  Reference  Framework  (TRF)  is  an  integrated  set  of  components  that  provide  a 
reusable  architecture  for  a  family  of  related  applications.  TRFs  should  be  capability- based  to 
maximize  employment  and  capability  insertion  on  multiple  platforms.  Limiting  the  number  of 
TRFs  will  increase  interoperability  and  reuse  opportunities,  leading  to  life  cycle  cost  savings. 

Maintaining  non- duplicative  TRFs  will  require  cooperative  interaction  and  create 
interdependencies  across  program  boundaries.  TRFs  are  dynamic  so  will  continue  to  evolve  as 
technology  dictates.  Configuration  management  and  attribute/characteristic  alignment  processes 
will  be  crucial  TRF  enablers.  To  develop  and  maintain  TRFs,  the  Naval  Enterprise  shall  take  the 
following  steps: 

1.  Analyze  existing  system  TRFs  and  develop  a  detailed  set  of  proven  Enterprise  attributes, 
including  standardized  specifications,  architectures,  data  models,  interoperability  protocols, 
and  software  development  tools; 

2.  Catalog  features  and  suitability  for  a  variety  of  platform  types; 

3.  Promote  tailor-able  open  standards  relative  to  TRF  attributes; 

4.  Coordinate  cross-program  TRF  implementations  to  reduce  duplication  through  transparency; 

5.  Identify,  publish,  and  manage  TRF  elements  necessary  to  move  programs  to  coordinated 
product  lines  and  S&T  investments  using  enterprise-level  TRF  attributes;  and 

6.  Require  PEOs  and  Systems  Commands  to  use  TRFs  for  all  development  unless  explicitly 
waived  by  ASN  (RD&A). 

Change  Acquisition  Processes  to  Adopt  OSA  TRFs 

Changes  must  be  made  to  current  Naval  acquisition  processes  to  allow  Enterprise  adoption  of 
OSA  TRFs.  The  Naval  Enterprise  shall  take  the  following  steps: 

1.  Charter  cross  PEO  groups  and  Communities  of  Interest  (COIs)  through  Program 
Management  Offices  (PMOs)  to  steer  the  development  of  common  TRFs,  applications,  and 
testing  strategies; 

2.  Identify  best  practices  and  collaborative  forums  to  increase  the  likelihood  of  transitioning 
maturing  technology  into  programs  of  record; 

3.  Change  guidance,  procedures,  and  instructions  to  require  preference  for  OSA  implementation 
and  systematic  reuse  for  cost  savings  across  system  life  cycles;  and 

4.  Insert  OSA  into  the  System  Engineering  Technical  Review  (SETR)  and  acquisition  program 
Gate  Review  processes. 

Require  and  Incentivize  Competition  throughout  Program  Life  Cycles 

The  Navy  values  innovation  and  lower  costs  at  all  acquisition  phases  (i.e.,  concept  development 
design,  build,  maintenance  and  upgrade)  and  system  levels  (i.e.,  component  system,  platform,  and 
system  of  systems).  The  Naval  Enterprise  shall  take  the  following  competition-focused  steps: 

1.  Create  contract  language  templates  for  use  in  contract  solicitations  at  the  platform, 
integrator,  system,  and  component  levels; 

2.  Develop  tools  and  methods  to  promote  competition  at  the  component  level  and  to 
objectively  measure  the  openness  of  development  environments; 

3.  Require  Program  Managers  to  evaluate  movement  away  from  monolithic  acquisitions  to 
multiple,  modular  acquisitions  enabled  by  OSA; 

4.  Require  Program  Managers  to  secure  and  exercise  data  rights  needed  to  ensure 
future  competition  for  sustainment  maintenance,  and  capability  insertion;  and 

5.  Establish  reward  mechanisms  for  programs  and  personnel  successful  in  achieving 
OSA  implementations  that  rapidly  integrate  innovation  and  lower  total 
ownership  costs. 


Open  Systems  Architecture  (OSA)  Strategy 
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Establish  Meaningful  Metrics  to  Assess  Progress 

Development  and  adoption  of  metrics  that  are  objective,  readily  obtained  (ideally  from  existing 
sources),  easy  to  interpret  and  actionable  to  enforce  desired  behaviors  (i.e.,  increased  competition, 
component  reuse  and  reduced  costs)  are  vital  to  the  OSA  strategy.  The  Naval  Enterprise  shall  take 
the  following  steps  to  implement  an  OSA  metrics  program: 

1.  Establish  a  set  of  metrics  for  use  in  assessing  the  Enterprise  value  of  new  capabilities; 

2.  Pilot  these  metrics  to  selected  COIs/Programs  of  Record  (PORs)  from  each  domain; 

3.  Update  metrics  based  on  these  pilots  for  application  across  the  Naval  Enterprise; 

4.  Implement  an  Enterprise  metrics  program  and  conduct  periodic  peer  review  assessments 
on  a  sampling  of  PORs  from  across  the  Enterprise;  and 

5.  Identify  patterns  of  strengths  and  weaknesses  in  Enterprise  OSA  implementation  and  apply 
remediation  throughout  program  life  cycles. 

Develop  the  OSA  Strategy  Execution  Guidebook 

The  Execution  Guidebook  will  contain  actionable  steps  for  each  implementation  phase  of 
the  OSA  strategy.  It  will  contain  recommended  changes  in  the  business  model  and  technical 
framework  elements  that  will  begin  by  improving  competition  and  ultimately  result  in  fewer 
programs  that  cost  less  and  deliver  capability  more  rapidly. 

Educate  the  Naval  Engineering,  Logistics,  and  Acquisition  Workforce 

The  success  of  the  OSA  Strategy  depends  heavily  on  a  competent,  innovative,  and  well  educated 
workforce.  The  Naval  Enterprise  must  produce  a  workforce  that  is  well- versed  in:  identifying  and 
managing  cross-domain  and  life  cycle  dependencies,  understanding  and  responding  to  adverse 
vendor  behaviors,  ensuring  that  competition  yields  the  desired  results,  and  incorporating  OSA  best 
practices  as  an  integral  part  of  program  management  The  Naval  Enterprise  shall  take  the  following 
steps  to  develop  an  OSA  workforce: 

1.  Target  timely  OSA  training  and  communication  to  optimize  acquisition  program  adoption; 

2.  Develop  training  and  communication  materials,  leveraging  existing  training  materials,  use 
cases,  and  delivery  mechanisms  when  possible; 

3.  Establish  OSA  transparency  mechanisms  to  enable  the  acquisition  workforce  to  become 
aware  of  opportunities  for  collaboration; 

4.  Work  with  the  Defense  Acquisition  University  (DAU)  to  develop  an  Acquisition  OSA 
Qualification  Standard; 

5.  Develop  training  materials  and  methodologies  to  train  the  non  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA)  Naval  workforce  involved  in  engineering,  logistics, 
and  program  management;  and 

6.  Establish  an  OSA  mentoring  program  for  acquisition  professionals. 


Assistant  Secretary  of  the  Navy  for  Research ,  Development,  and  Acquisition 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-684- 


Computer-Aided  Process  and  Tools  for  Mobile  Software 

Acquisition 


Christopher  Bonine — Bonine  is  a  lieutenant  in  the  United  States  Navy.  He  is  currently  assigned  to 
the  Navy  Cyber  Defense  Operations  Command  in  Norfolk,  VA.  He  has  served  as  information  warfare 
officer  onboard  the  USS  Sampson  and  as  N51  division  officer  at  the  Navy  Cyber  Warfare 
Development  Group.  His  current  interests  are  in  the  development  and  implementation  of  cyber 
security  policy.  Bonine  has  a  master’s  in  computer  science  from  the  Naval  Postgraduate  School. 
[cbbonine@nps.edu] 

Man-Tak  Shing — Shing  is  an  associate  professor  at  the  Naval  Postgraduate  School.  His  research 
interests  include  the  engineering  of  software  intensive  systems.  He  is  on  the  program  committees  of 
several  software  engineering  conferences.  He  was  the  program  co-chair  of  the  Rapid  System 
Prototyping  Workshop  in  2004  prior  to  being  the  general  co-chair  for  the  symposium  in  2008.  He  also 
served  as  the  program  co-chair  of  the  IEEE  System  of  Systems  Engineering  Conference  in  2010  and 
2011.  He  received  his  PhD  in  computer  science  from  the  University  of  California,  San  Diego,  and  is  a 
senior  member  of  IEEE,  [shing@nps.edu] 

Thomas  W.  Otani — Otani  is  an  associate  professor  of  computer  science  at  the  Naval  Postgraduate 
School.  His  main  research  interests  include  object-oriented  programming,  mobile  and  web  application 
development,  and  database  design.  He  received  his  PhD  in  computer  science  from  the  University  of 
California,  San  Diego,  in  1983.  [twotani@nps.edu] 

Abstract 

Mobile  devices  have,  in  many  ways,  replaced  traditional  desktops  in  usability,  usefulness, 
and  availability.  Many  companies  are  scrambling  to  develop  enterprise  strategies  to  provide 
mobile  devices  and  application  support  for  their  employees,  and  the  DoD  is  taking  the  point  in 
the  federal  government’s  campaign  to  deploy  mobile  devices.  A  successful  DoD  mobile 
software  acquisition  program  requires  efficient  and  effective  means  to  assure  the  proper 
functioning  of  the  applications.  As  the  majority  of  future  mobile  apps  will  be  developed  by 
small  companies  (or  crowdsourcing  individuals)  and  have  relatively  short  development 
cycles,  a  traditional  software  verification  process  that  relies  on  the  testing  of  source  code  is 
not  effective  for  vetting  mobile  apps.  The  paper  presents  a  new  approach  for  vetting  mobile 
software.  It  allows  subject  matter  experts  to  specify  desirable  and  undesirable  behaviors  of 
the  mobile  apps  as  executable  statecharts  and  to  verify  the  target  software  by  running  the 
automatically  generated  statechart  code  against  the  execution  trace  of  the  mobile  apps  using 
log  file-based  runtime  verification.  A  case  study  of  formally  specifying,  validating,  and 
verifying  a  set  of  requirements  for  an  iPhone  application  that  tracks  the  movement  of  the 
iPhone  user  is  used  to  demonstrate  the  new  approach. 

Introduction 

In  an  April  23,  2012,  blog  post,  analyst  Frank  E.  Gillett  of  Forrester  Research 
predicted  that  “tablets  will  become  our  primary  computing  device”  in  the  near  future,  with 
“global  tablet  sales  to  reach  375  million  units,  with  one-third  purchased  by  businesses  and 
two-fifths  (or  40  percent)  by  emerging  markets”  by  2016  (Gillett,  2012).  Many  companies  are 
scrambling  to  develop  enterprise  strategies  to  provide  mobile  devices  and  application 
support  for  their  employees,  and  the  DoD  “is  taking  the  point  in  the  federal  government’s 
campaign  to  deploy  mobile  devices”  (Kenyon,  2012a).  The  Defense  Information  Systems 
Agency  (DISA)  has  opened  a  program  office  and  issued  a  request  for  information  to  solicit 
ideas  from  industry  for  ways  to  provide  the  mobile  device  management  (MDM)  services  and 
to  run  an  applications  store  (Kenyon,  2012b),  and  the  Army  has  established  the  Army 
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Software  Marketplace,  a  prototype  online  storefront  for  Army-wide  distribution  of  mobile 
software. 

As  the  DoD  is  charging  forward  with  its  mobile  programs,  it  must  find  ways  to 
address  its  concerns  in  security,  authentication,  and  logistics  in  managing  and  deploying  the 
rapidly  growing  number  of  mobile  applications  and  devices  with  varying  degrees  of  access 
across  the  DoD  enterprise.  The  Space  and  Naval  Warfare  (SPAWAR)  Atlantic  System 
Center  is  working  with  DISA  and  the  National  Institute  of  Standards  and  Technology  (NIST) 
to  provide  warfighters  with  access  to  unclassified  information  from  their  handheld  devices 
via  the  cloud-based  mobility-as-a-service,  and  the  recent  adoption  of  a  hardened  kernel  for 
the  Android  mobile  operating  system  is  another  major  step  towards  providing  a  secure  base 
for  the  development  of  trustworthy  mobile  software.  Moreover,  the  DoD  needs  an  efficient 
and  effective  process  to  ensure  the  proper  functioning  of  the  mobile  software  (commonly 
referred  to  as  mobile  apps),  so  that  the  software  does  what  it  promises  to  do  and  does  so 
without  hidden  or  emergent  malicious  behaviors. 

Mobile  apps  shrink  the  software  programs  that  were  once  only  available  on  a 
desktop  computer,  making  them  usable  on  smart  phones  and  mobile  devices.  The  app 
market  has  been  growing  at  an  unprecedented  rate.  The  app  world,  which  consisted  of 
8,000  Apple  titles  in  2008,  had  reached  1  million  titles  in  201 1  (Freierman,  201 1 ).  As  the 
majority  of  mobile  apps  are  developed  by  small  companies  (or  crowdsourcing  individuals) 
and  have  relatively  short  development  cycles,  traditional  software  verification  processes  that 
rely  on  testing  of  source  code  are  not  effective  for  vetting  mobile  software.  The  DoD  needs 
better  means  to  ensure  the  proper  functioning  of  mobile  apps  without  source  code  or  other 
detailed  information  about  the  software’s  implementation. 

This  paper  presents  a  new  approach  for  vetting  mobile  software.  It  allows  subject 
matter  experts  to  specify  desirable  and  undesirable  behaviors  of  the  mobile  apps  as 
executable  statecharts  and  to  verify  the  target  software  by  running  the  automatically 
generated  statechart  code  against  the  execution  trace  of  the  mobile  apps  using  log  file- 
based  runtime  verification. 

The  rest  of  the  paper  is  organized  as  follows.  The  V&V  of  Mobile  Apps  section 
provides  a  summary  of  the  current  state  of  verification  and  validation  (V&V)  of  mobile  apps. 
The  Formal  Specification  and  Validation  of  Mobile  Apps  section  presents  an  overview  of 
statechart  assertions,  our  formal  specification  language  of  choice,  and  the  proposed 
computer-aided  process  for  the  V&V  of  mobile  apps.  The  section  Case  Study  presents  a 
case  study  involving  the  formal  specification,  validation,  and  verification  of  a  set  of 
requirements  for  an  iPhone  application  that  tracks  the  movement  of  the  iPhone  user.  The 
last  section  is  the  conclusion,  which  provides  a  summary  and  draws  some  conclusions. 

The  V&V  of  Mobile  Apps 

Verification  and  Validation  (V&V)  is  a  software  evaluation  process  to  ensure  proper 
and  expected  operation.  As  stated  in  Michael,  Drusinsky,  Otani,  and  Shing  (2011), 

Verification  refers  to  activities  that  ensure  the  product  is  built  correctly  by 
assessing  whether  it  meets  its  specifications.  Validation  refers  to  activities 
that  ensure  the  right  product  is  built  by  determining  whether  it  meets 
customer  expectations  and  fulfills  specific  user-defined  intended  purposes. 

Simply  stated,  the  purpose  of  V&V  is  to  ensure  the  software  does  what  it  is  required  to  do, 
and  nothing  more. 
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Difficulties  in  Testing  Mobile  Apps 

New  mobile  devices,  especially  phones,  have  such  short  development  times  that  the 
devices  have  barely  been  on  the  market  long  enough  to  work  out  existing  bugs  before  the 
new  device  with  new  software  is  ready  to  release.  As  an  example,  Apple  releases  a  new 
iPhone  model  every  year,  and  has  developed  six  generations  of  iOS.  The  Android  operating 
system  had  eight  versions  in  three  years.  This  high  turnover  of  mobile  devices  is  created  not 
only  by  demand  and  competition,  but  also  capability  increases  of  computing  power,  battery 
life,  and  screen  size.  As  new  capabilities  are  added  to  the  devices  and  applications  in  each 
development  cycle,  new  automated  V&V  techniques  are  needed  to  keep  up  with  the  fast 
pace  of  mobile  application  development. 

Additional  difficulties  in  the  testing  of  mobile  applications  are  due  to  limitations  of  the 
hardware.  At  this  time,  other  than  operating  system  tasks,  iPhone  can  only  run  a  single 
application  at  a  single  point  in  time.  The  purpose  is  to  conserve  the  limited  computing  power 
of  the  device  as  well  as  reduce  power  consumption.  The  negative  aspect  is  that  there  is  little 
or  no  application  interaction  on  a  single  device.  This  prevents  useful  testing  applications 
from  running  on  mobile  devices  to  analyze  the  real-time  behavior  of  applications.  Even  if 
such  an  ability  were  possible,  the  small  screen  size  would  create  difficulties  in  analyzing  the 
data  while  on  the  device.  Android  devices  have  the  ability  for  third-party  developers  to 
create  multiprocessing  applications,  which  could  allow  analytics  to  be  conducted  directly  on 
the  device,  but  the  same  screen  size  limitation  would  impede  analysis  of  the  data  (see 
Muccini,  Francesco,  &  Esposito  [2012]  for  a  detailed  discussion  of  the  challenges  in  testing 
mobile  apps.) 

These  limitations  make  testing  done  off  the  device  more  amenable.  There  are  two 
possible  options:  use  device-specific  emulators,  or  use  specially  altered  software  code  to 
allow  offloading  of  real  data  from  the  device  onto  a  computer  for  analysis.  While  the 
emulators  will  do  a  good  job  creating  a  proper  environment  to  test  an  application,  it  has  the 
limitation  of  being  stuck  in  place,  and  does  not  recreate  the  ever-changing  environment  in 
which  mobile  devices  exist.  The  other  method  could  potentially  include  such  a  robust 
environment;  the  currently  existing  techniques  require  a  cable  connection  to  a  computer, 
tethering  the  mobile  device  to  an  immobile  one.  The  current  techniques  also  require  an 
instrumented  version  of  the  original  code  to  provide  a  mechanism  to  offload  the  required 
information  to  properly  evaluate  the  operation  of  the  application. 

Current  Solutions  to  V&V  of  Mobile  Apps 

Monkeyrunner  enables  the  writing  of  unit  tests  to  test  software  at  a  functional  level 
(“Monkeyrunner,”  n.d.).  Monkeyrunner  uses  Python  to  run  testing  code  on  one  or  more 
devices,  or  an  emulator.  It  can  send  commands  and  keystrokes,  and  record  screenshots. 
Monkeyrunner  allows  for  the  repetition  of  test  results,  but  element  location  in  the  recorded 
screenshots  is  the  basis  for  comparing  two  test  results.  This  limits  comparisons  to  a  single 
screen  size. 

Android  Robotium  is  a  Java-based  tool  for  writing  unit  tests  (“User  Scenario  Testing,” 
n.d.).  Similar  to  Monkeyrunner,  it  is  designed  to  run  as  a  black-box  testing  tool  and  can  run 
as  an  emulator,  as  well  as  run  on  the  actual  device,  although  it  is  limited  to  a  single  device. 
Robotium  allows  for  testing  of  pre-install  software  as  well.  The  big  difference  between 
Robotium  and  Monkeyrunner  is  that  Robotium  has  a  more  robust  test  result  comparison. 
Rather  than  using  a  location-based  method,  Robotium  uses  identifiers  to  recognize 
elements.  This  allows  devices  of  different  types  and  sizes  to  be  compared  to  ensure 
consistency. 
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Lesspainful.com  provides  a  way  for  customers  to  run  software  and  unit  tests  on 
physical  devices  without  the  cost  of  owning  the  devices  (Lesspainful  Device  Lab,  n.d.).  The 
customers  use  the  programming  language  Cucumber  to  write  an  English  description  of  the 
test  they  would  like  to  run  on  their  software.  Once  the  devices  to  be  tested  on  are  chosen, 
the  tests  are  automated  in  a  cloud-like  system  with  results  from  each  mobile  device 
presented  to  the  customer  to  allow  for  easy  comparison. 

Testquest  10  is  a  software  suite,  created  by  Bsquare,  which  enables  unit  tests  in  a 
device  emulator  and  enables  the  collaboration  of  geographically  dispersed  teams  (Bsquare, 
2003).  It  utilizes  an  extensive  use  of  image  recognition  to  determine  device  state  as  well  as 
the  location  of  applications  and  features  on  the  screen.  An  interesting  feature  is  that  if  the 
GUI  design  is  changed  and  an  application  or  feature  is  moved  from  one  location  to  another, 
this  suite  is  able  to  locate  and  use  the  feature. 

Bo,  Xiang,  and  Xiaopeng  (2007)  introduced  an  approach  for  testing  a  device  and 
software  by  using  what  they  called  sensitive-events.  Their  approach  reduces  the  need  for 
screenshot  comparisons  by  capturing  these  events,  such  as  inbox  full,  to  determine  state 
change.  The  software  will  then  evaluate  these  state  changes  and,  if  the  events  indicate 
desired  conditions,  the  tests  will  continue. 

All  of  the  aforementioned  software  tools  are  for  testing  an  application  to  ensure 
proper  functionality  and  operations.  What  they  are  missing  is  the  ability  to  map  the  operation 
of  the  phone  directly  to  a  set  of  requirements.  The  above  tools  all  require  some  form  of 
script  writing,  which  can  lead  to  missing  software  test  cases.  When  writing  scripts  to  cover 
unit  tests,  the  programmer  must  understand  the  requirements  and  determine  boundary 
(edge)  cases  in  order  to  properly  test  for  them.  The  tools  are  also  limited  in  their  ability  to 
handle  context-aware  features.  Another  limitation  is  that,  due  to  the  limitation  of  the 
hardware  and  the  software  testing  suites,  only  one  application  at  a  time  can  be  tested. 

Delamaro,  Vincenzi,  and  Maldonado  (2006)  used  an  extension  to  the  JaBUTi,  called 
JaBUTi/ME.  The  extension  takes  JaBUTi,  which  is  a  Java  byte  code  analysis  tool,  and  adds 
the  ability  to  run  instrumented-code  on  a  mobile  device  that  creates  trace  data,  and  then 
pass  the  trace  data  to  a  desktop  computer  for  analysis.  By  using  a  method  of  creating  trace 
data,  this  solution  is  conceptually  similar  to  the  idea  presented  in  this  paper.  However,  this 
method  still  requires  test  cases  to  be  manually  written  to  evaluate  the  resulting  trace  file. 
Additionally,  as  stated  by  the  authors,  the  code  instrumentation  would  vary  based  on  the 
hardware  device  the  code  is  being  tested  on.  This  is  due  to  the  potential  differences  in 
network  connectivity  needed  to  transmit  the  trace  data  back. 

Formal  Specification  and  Validation  of  Mobile  Apps 

Michael  et  al.  (2011)  pointed  out  that 

Software  engineers  have  become  competent  at  verification:  we  can  build 
portions  of  systems  to  their  applicable  specifications  with  relative  success. 
However,  we  still  build  systems  that  don’t  meet  customers’  expectations  and 
requirements.  This  is  because  people  mistakenly  combine  V&V  into  one 
element,  treating  validation  as  the  user’s  operational  evaluation  of  the 
system,  resulting  in  the  discovery  of  requirement  errors  late  in  the 
development  process,  when  it’s  costly,  if  not  impossible,  to  fix  those  errors 
and  produce  the  right  product. 

Hence,  first  and  foremost,  we  need  a  means  for  analysts  to  describe  the  desirable 
and  undesirable  behaviors  of  the  mobile  apps.  Typically,  the  requirements-discovery 
process  begins  with  constructing  scenarios  involving  the  system  and  its  environment.  From 
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these  scenarios,  analysts  informally  express  their  understanding  of  the  system’s  expected 
behavior  or  properties  using  natural  language  and  then  translate  them  into  a  specification. 
Specification  based  on  natural  language  statements  can  be  ambiguous.  For  example, 
consider  the  following  requirement  for  a  project  management  software:  The  software  shall 
generate  a  project  status  report  once  every  month.  Will  the  software  meet  the  customer’s 
expectation  if  it  generates  one  report  each  calendar  month?  Does  it  matter  if  the  software 
generates  one  report  in  the  last  week  of  May  and  another  in  the  first  week  of  June?  What 
happens  if  a  project  lasts  only  15  days?  Does  the  software  have  to  generate  at  least  one 
report  for  such  a  project? 

Research  has  shown  that  formal  specifications  and  methods  help  improve  the  clarity 
and  precision  of  requirements  specifications  (Easterbrook  et  al.,  1998).  However,  formal 
specifications  are  useful  only  if  they  match  the  true  intent  of  the  customer’s  requirements. 
Because  only  the  subject  matter  expert  (SME)  who  supplied  the  requirements  can  answer 
these  questions,  the  analyst  must  validate  his  or  her  own  cognitive  understanding  of  the 
requirements  with  the  SME  to  ensure  that  the  specification  is  correct.  For  example,  consider 
the  security  requirement  R1:  If  there  are  more  than  two  invalid  logins  within  any  15-second 
interval,  then  the  mobile  device  will  remain  unavailable  for  10  minutes.  Whether  the  scenario 
shown  in  Figure  1  violates  R1  depends  on  the  interpretation  of  the  starting  time  of  10-minute 
timeout  interval. 
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Figure  1.  Example  of  Requirements  Ambiguity 


The  best  way  to  validate  and  disambiguate  complex  behavioral  requirements  is  to 
walk  through  the  different  scenarios  with  the  stakeholders  and  ask  them  to  confirm  or  clarify 
the  requirements  analyst’s  cognitive  understanding  of  the  natural  language  requirements. 
Drusinsky,  Shing,  and  Demir  (2007)  proposed  the  iterative  process  for  assertion  validation 
shown  in  Figure  2.  This  process  encodes  requirements  as  Unified  Modeling  Language 
(UML)  statecharts  augmented  with  Java  action  statements  and  validates  the  assertions  by 
executing  a  series  of  scenarios  against  the  statechart-generated  executable  code  to 
determine  whether  the  specification  captures  the  intended  behavior. 
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Figure  2.  Iterative  Process  for  Assertion  Validation 

(based  on  Drusinsky,  Shing,  &  Demir,  2007) 


Statechart  Assertions 

A  statechart  assertion  is  a  U ML  statechart-based  formal  specification  for  use  in 
prototyping,  runtime  monitoring,  and  execution-based  model  checking  (Drusinsky,  2011).  It 
extends  the  Harel  statechart  formalism  (Harel,  1987)  and  is  supported  by  StateRover,  a 
plug-in  for  the  Eclipse  integrated  development  environment  (IDE;  see 
www.timerover.com/staterover.pdf).  StateRover  provides  support  for  design  entry,  code 
generation,  and  visual  debug  animation  for  UML  statecharts  combined  with  flowcharts. 


The  statechart  assertion  extends  Harel  statecharts  by  adding  a  bSuccess  Boolean 
flag  and  by  enabling  non-determinism.  Statechart  assertions  are  formulated  from  an  external 
observer’s  perspective.  Though  the  bSuccess  Boolean  is  a  simple  mechanism,  it  is 
instrumental  in  determining  if  an  assertion  ever  fails.  The  Boolean  indicates  whether  the 
assertion  was  violated  by  the  system  being  analyzed.  A  statechart  assertion  assumes  the 
requirement  it  is  based  on  is  met  ( bSuccess  =  true),  and  it  will  retain  that  assumption  unless 
a  sequence  of  events  leading  to  the  violation  of  the  requirement  specified  by  the  statechart 
assertion  is  observed.  Once  an  assertion  fails  (i.e.,  reaches  an  error  state),  bSuccess 
becomes  false  and  will  stay  false  for  the  remainder  of  the  execution.  Since  the  statecharts 
are  simple,  it  is  easy  to  identify  the  assertion  that  failed  and  the  cause. 
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Figure  3  shows  a  statechart  assertion  for  the  requirement  R1,  where  the  10-minute 
interval  starts  immediately  at  the  detection  of  the  third  invalidLogin  event  within  a  15-second 
interval  according  to  the  analyst’s  interpretation  of  the  natural  language  requirement.  The 
statechart  is  written  from  the  standpoint  of  an  observer,  who  is  interested  in  the  proper 
sequencing  of  two  system  events:  invalidLogin  and  deviceUnlock.  It  uses  two  timers  to  keep 
track  of  the  timing  constraints  in  R1 .  Starting  out  in  the  Init  state,  the  statechart  transitions  to 
the  flowchart-action  box  StartTimer  when  it  observes  an  invalidLogin  event.  It  increments 
the  counter  nCnt  and  starts  the  15-second  timer,  and  then  checks  to  see  if  the  counter  nCnt 
exceeds  2.  If  nCnt  <  2,  it  enters  the  Count  state.  Whenever  the  statechart  observes  an 
invalidLogin  event  in  the  Count  state,  it  increments  the  counter  nCnt  and  then  checks  to  see 
if  the  counter  nCnt  exceeds  2.  The  statechart  will  remain  in  the  Count  state  until  either  the 
15-second  timer  expires,  or  until  nCnt  >  2.  If  nCnt  >  2,  the  statechart  enters  the  LockDevice 
state  and  starts  the  10-minute  timer.  The  statechart  will  remain  in  the  LockDevice  state  until 
either  the  10-minute  timer  expires,  or  until  it  observes  a  deviceUnlock  event.  It  the  statechart 
observes  a  deviceUnlock  event  in  the  LockDevice  state,  it  enters  the  Error  state.  The  entry 
action  for  the  Error  state  sets  bSuccess  to  false,  meaning  that  the  requirement  R1  has  been 
violated. 

The  StateRover  supports  the  specification  of  complex  requirements  using  non- 
deterministic  statecharts.  While  deterministic  statechart  assertions  suffice  for  the 
specification  of  many  requirements,  theoretical  results  show  that  non-deterministic 
statecharts  are  exponentially  more  succinct  than  deterministic  Harel  statecharts  (Drusinsky 
&  Harel,  1994).  Non-deterministic  statechart  assertions  provide  a  very  intuitive  way  for 
designers  to  specify  behaviors  involving  a  sliding  time  window.  In  the  statechart  assertion 
shown  in  Figure  3,  there  is  an  apparent  next-state  conflict  when  an  event  invalidLogin  is 
observed  in  the  Init  state.  StateRover  uses  a  special  code  generator  to  create  a  plurality  of 
state-configuration  objects  for  non-deterministic  statechart  assertions,  one  per  possible 
computation  in  the  assertion  statechart.  Non-deterministic  statechart  assertions  use  an 
existential  definition  of  the  isSuccess  method,  where  if  there  exists  at  least  one  state- 
configuration  that  detects  an  error  (assigns  bSuccess  =  false),  then  the  isSuccess  method 
for  the  entire  non-deterministic  assertion  returns  false.  Likewise,  terminal  state  behavior  is 
existential;  if  at  least  one  state  configuration  is  in  a  terminal  state,  then  the  non-deterministic 
statechart  assertion  wrapper  considers  itself  to  be  in  a  terminal  state. 
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For  example,  the  statechart  assertion  in  Figure  3  will  generate  four  state- 
configuration  objects  for  the  test  scenario  shown  in  Figure  4  at  runtime,  one  for  each 
invalidLogin  event.  The  state-configuration  object  that  starts  with  the  second  invalidLogin 
event  will  end  up  in  the  Error  state,  causing  the  isSuccess  method  to  return  false  to  the  test 
driver. 
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Figure  4.  An  Exception  Test  Scenario  for  the  Statechart  Assertion  R1 


Validation  of  Statechart  Assertions 

StateRover’s  Code  generator  generates  a  Java  class  R1  for  the  statechart  assertion 
file.  The  generated  code  is  designed  to  work  with  the  JUnit  Java  testing  framework  (Beck  & 
Gamma,  1998;  see  Figure  5). 


Figure  5.  Validating  Statechart  Assertion  via  Scenario-Based  Testing 


To  assure  that  the  statechart  assertion  works  as  specified  in  R1,  we  test  its  behavior 
using  the  JUnit  test  cases  corresponding  to  the  different  scenarios  shown  in  Figure  6  and 
the  one  shown  in  Figure  4. 
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Figure  6.  Test  Scenarios  for  the  Statechart  Assertion  R1 


Test  Scenarios  1  and  2  in  Figure  6  represent  two  typical  “happy”  scenarios.  Test 
Scenario  1  expects  the  system  to  detect  the  three  invalidLogin  events  within  a  15-second 
interval  and  then  lock  the  device  for  10  minutes.  Test  Scenario  2  expects  the  system  to  keep 
the  device  open  since  it  only  observes  two  invalidLogin  events  within  a  15-second  interval. 
Test  Scenario  3  represents  an  exception  scenario,  where  the  system  allows  the  device  to  be 
unlocked  too  early,  causing  the  statechart  assertion  to  enter  the  Error  state,  thereby 
signaling  that  the  assertion  detected  a  requirement  violation. 


Log  File-Based  Runtime  Verification  of  Mobile  Apps 

Alves,  Drusinsky,  Michael,  and  Shing  (2011)  presented  an  end-to-end  process  that 
begins  with  a  system  requirement  as  a  natural  language  specification,  followed  by  the 
creation  and  computer-aided  validation  of  UML  statechart-formal  specification  assertions, 
and  ending  with  the  log  file-based  runtime  verification  of  the  target  system.  These  log  files 
were  executed  as  JUnit  tests  against  the  assertions.  They  applied  the  process  to  the 
specification,  validation,  and  verification  (SV&V)  of  the  critical  time-constrained  requirements 
of  the  Brazilian  Satellite  Launcher  flight  software,  and  uncovered  several  inaccuracies  in  the 
requirements  understanding  and  implementation. 


Computer-Aided  Process  for  the  V&V  of  Mobile  Apps 

We  shall  apply  similar  process  to  conduct  the  V&V  of  mobile  apps,  which  consists  of 
the  following  steps: 

1 .  Subject  matter  experts  determine  the  properties  of  interest  and  the  metrics  to 
verify/measure  those  properties  in  the  lab. 

2.  The  properties  are  then  expressed  precisely  as  statechart  assertions,  whose 
correctness  is  validated  via  runtime  verification. 

3.  The  mobile  devices  and  applications  are  then  instrumented,  if  needed,  for 
data  collection  and  log  file  generation. 

4.  The  instrumented  codes  are  deployed  to  the  field  via  mobile  apps  downloads. 
Metric  data  are  collected  in  log  files  while  the  mobile  devices  are  being  used 
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in  the  tactical  environment,  and  the  log  files  are  uploaded  back  to  the  lab 
while  the  mobile  devices  are  being  recharged. 

5.  The  log  files  are  then  converted  into  JUnit  tests,  and  the  tests  are  run  against 
the  statechart  assertions  in  the  lab.  The  test  results  are  analyzed  and 
reported. 

Using  log  files  produced  by  mobile  apps  brings  two  benefits:  (1)  it  captures  the 
behavior  of  the  application  on  an  actual,  physical  device  and  (2)  the  data  contained  in  the 
file  will  represent  the  behavior  of  the  application  as  it  executes.  Log  files  collected  by  the 
application  in  execution  on  a  device  that  is  fully  mobile  hold  data  that  is  representative  of  the 
expected  normal  operation  of  the  application.  Therefore,  we  can  analyze  the  log  files  to 
determine  if  the  behavior  was  correct  based  on  the  requirements.  As  demonstrated  in  the 
next  section,  we  do  not  need  to  instrument  the  mobile  device  or  its  software  if  the  events  of 
interest  are  derivable  from  the  output  data  of  the  mobile  apps. 

Case  Study 

The  case  study  involves  a  smartphone  application  that  uses  a  GPS  to  track  the 
location  and  speed  of  a  person  in  motion.  A  log  of  the  collected  GPS  data  must  be  kept  in 
the  smartphone  until  it  can  be  uploaded  to  a  server  via  a  Wi-Fi  connection.  GPS  applications 
can  consume  a  lot  of  power  and  storage  space  and  since  mobile  devices  have  limited 
amounts  of  both,  minimizing  the  consumption  of  both  is  important. 

Due  to  the  limited  available  storage  space  on  the  mobile  device,  we  must  minimize 
the  amount  of  GPS  data  stored.  The  method  chosen  to  accomplish  this  is  to  adjust  the  rate 
at  which  the  GPS  updates  occur  to  be  based  on  the  speed  at  which  the  user  is  traveling.  An 
additional  requirement  is  that  the  log  file  must  be  able  to  be  transmitted  from  the  device  to  a 
server  by  a  Wi-Fi  connection  only.  This  is  because  many  of  the  users  will  not  have  wired 
connectors  for  the  devices.  If  at  any  point  Wi-Fi  connectivity  is  lost  and  there  is  an  active 
transmission,  it  must  be  terminated.  The  application  has  a  limit  of  30  seconds  to  transmit  the 
log  file,  after  which,  if  not  successful,  the  user  must  be  notified  of  the  failed  transmission 
within  five  seconds.  Additionally,  a  log  file  must  not  be  transmitted  within  one  hour  of  a 
previous  log  transmission.  Both  the  use  of  a  time-limited  transmission  window  for  the  log  file 
as  well  as  an  infrequent  upload  of  the  log  file  will  aid  in  reducing  the  amount  of  power  and 
bandwidth  the  application  consumes. 

Specification  and  Validation  of  the  Statechart  Assertions 

When  a  user  is  traveling  at  a  slow  speed  like  walking,  frequent  updates  are 
unnecessary  because  significant  distance  changes  do  not  happen  quickly.  If  the  user  is 
traveling  at  a  faster  pace,  then  more  updates  allow  for  more  consistent  tracking.  When  the 
user  is  traveling  at  less  than  or  equal  to  two  meters  per  second,  the  application  should 
average  five  seconds  or  more  per  update.  This  is  approximately  the  walking  speed  of  a 
human  (Carey,  2005).  If  the  user  is  traveling  at  greater  than  two  meters  per  second,  but  less 
than  or  equal  to  five  meters  per  second,  then  there  must  be  an  average  of  between  two  and 
five  seconds  between  updates.  This  is  considered  running  speed.  If  traveling  greater  than 
five  meters  per  second,  then  there  must  be  an  average  of  less  than  two  seconds  between 
updates.  This  is  driving  speed. 

We  decided  to  use  an  average  of  seconds  between  updates  due  to  the  typically  less- 
than-accurate  GPS  data  provided  by  mobile  devices.  A  requirement  for  an  average  over  a 
minimum  of  five  GPS  update  events  will  be  included  to  reduce  the  effects  of  any  lack  of 
precision  in  the  GPS  data  from  the  mobile  device.  Table  1  lists  the  requirements. 
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Table  1.  Speed-Based  Requirements 


Speed  Category 

Average  Speed  (x)  in 
meters  per  second 

Expected  Time  between  Updates  (y) 
in  Seconds  per  Update 

Walking 

x<=2 

5<=y 

Running 

2<x<=5 

2<=y<5 

Driving 

5<x 

y<2 

Drusinsky,  Michael,  and  Shing  (2007)  stated  that  a  model-based  specification  that 
uses  a  single,  intertwined  representation  of  the  software  requirements  (e.g.,  as  a  single 
statechart)  can  become  complex  and  difficult  to  understand  due  to  the  interaction  of  each 
requirement  with  others.  They  advocated  the  use  of  assertion-based  specification,  which 
allows  the  requirements  to  be  decomposed  into  their  simplest  forms,  and  then  create  a 
formal  representation  (e.g.,  a  statechart  assertion)  for  each  requirement.  This  decomposition 
allows  a  one-to-one  connection  between  a  statechart  assertion  and  a  customer  requirement. 
A  significant  benefit  of  this  connection  is  that  it  simplifies  the  development,  analysis,  and 
testing  of  the  statechart  assertions.  Other  benefits  include  the  following: 

1 .  Reduction  of  the  statechart  assertion  complexity:  Since  the  complexity  of  the 
statechart  assertions  is  minimized,  the  statechart  assertions  are  much  easier 
to  test  for  correctness. 

2.  The  one-to-one  connection  between  a  statechart  assertion  and  a  customer 
requirement  simplifies  the  changes  that  need  to  be  made  to  the  assertions 
when  the  requirements  change. 

3.  Statechart  assertions  can  be  made  to  represent  a  test  for  both  negative  and 
positive  behaviors,  whereas  a  model-based  specification  usually  only 
captures  positive  behaviors. 

4.  Tracing  unexpected  behaviors  to  the  one  or  more  requirements  that  they 
violate  is  simpler  because  there  is  a  one-to-one  mapping. 

Hence,  we  refine  the  speed-based  GPS  Update  requirement  into  three  requirements. 
Figures  7,  8,  and  9  show  the  statechart  assertions  for  each  of  the  three  speed  categories  of 
the  speed-based  GPS  Update  requirement. 
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Figure  8.  Statechart  Assertion  for  Speeds  Between  2  and  5  Meters  per  Second 
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Figure  9.  Statechart  Assertion  for  Speeds  Greater  Than  5  Meters  per  Second 


Figure  10  shows  the  statechart  assertion  for  the  requirement  that  a  log  file  can  only 
be  transmitted  when  the  device  is  connected  to  a  Wi-Fi  access  point.  Note  that  this 
statechart  assertion  only  covers  the  requirement  that  a  transmission  cannot  start  when  not 
connected  to  Wi-Fi,  but  does  not  capture  the  requirements  that  log  files  cannot  be 
transmitted  within  an  hour  of  each  other,  nor  does  it  cover  what  needs  to  be  done  when  the 
Wi-Fi  connection  is  lost  during  a  transmission.  We  chose  to  capture  the  latter  with  three 
other  statechart  assertions  (Figures  11,  12,  and  1 3),  thus  simplifying  the  complexity  of  each 
statechart  assertion. 
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Figure  10.  Statechart  Assertion  for  Wi-Fi-Only  Transmission 
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Figure  11.  Statechart  Assertion  Limiting  Log  File  Transmission  Time  to  30  Seconds 


Figure  12.  Five  Seconds  to  Notify  User  of  Transmission  Failure 
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Figure  13.  One-Hour  Time  Out  Between  Successive  Log  File  Transmissions 

We  tested  each  of  the  above  statecharts  with  different  scenarios  to  ensure  that  they 
correctly  capture  the  intent  of  the  natural  language  requirements. 
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Log  File  Preprocessing  and  Runtime  Verification 

The  GPS  application  generates  log  files  with  the  data  format  shown  in  Figure  14, 
which  is  different  from  those  required  by  StateRover,  like  those  shown  in  Figure  15. 

WIFI_CONN  @  11/30/2012  01:07:44  PM 
WIFI  DISCONNECT  @  11/30/2012  01:07:45  PM 


Figure  14.  GPS  Application  Generated  Data  Format 


<event> 

<sig><! [CDATA[wif iConn] ]></sig> 

<time  lang="c"  unit="sec"  val="1354309664"  /> 
</event> 

<event> 

<sig><! [CDATA[wif iDisconn] ]></sig> 

<tirce  lang=”c"  unit="sec"  val="1354309665"  /> 
</event> 


Figure  15.  StateRover  Required  Log  File  Format 

In  order  to  test  the  log  file  produced  by  the  GPS  application  against  the  statechart 
assertions,  we  need  to  convert  the  original  log  into  a  log  that  can  be  read  by  the  StateRover 
tool.  We  developed  a  Python  script  to  convert  the  application  log  file  into  what  we  shall  call  a 
StateRover  log  file.  Using  StateRover’s  log  file-to-JUnit  converter,  the  StateRover  log  file 
was  imported  into  the  off-line  verification  environment  and  converted  into  an  equivalent 
JUnit  Java  class.  This  class  contained  the  log  file-based  verification  test  for  the  statechart 
assertions.  Using  StateRover’s  namespace  mapping  tool,  we  created  a  namespace 
mapping  that  linked  the  JUnit  Java  class’s  name  space  (events  as  defined  in  the  log  files)  to 
the  assertion  repository’s  namespace  (events  of  the  statechart  assertions).  The 
StateRover’s  namespace  mapping  in  Figure  16  depicts  on  the  left-side  tree  (denoted  the 
source  tree)  events  taken  from  a  log  file  and,  on  the  right-side  tree  (denoted  the  target  tree), 
events  from  all  assertions  in  the  assertion  repository.  Connections  between  the  source  and 
the  target  trees  can  be  done  manually  using  the  user  interface,  or  automatically  using  a 
built-in  matching  algorithm. 
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Figure  16.  Namespace  Mapping  for  Runtime  Verification 


Once  this  is  complete,  the  test  can  be  run  by  pressing  the  Run  button  in  the  toolbar. 
Figure  17  shows  the  desired  result  after  testing  one  or  more  statechart  assertions.  If  an 
assertion  failure  exists  (i.e. ,  a  bSuccess  variable  in  one  of  the  assertions  was  set  to  false), 
the  statechart  assertion  where  it  occurs  will  be  listed  on  the  left  side  under  the  header 
Statechart  Assertion  Failures. 


To  validate  the  correct  operation  of  the  statechart  assertions,  we  manually  generated 
some  log  files  containing  errors.  The  log  file  in  Figure  18  is  an  example  snippet  of  such  a  log 
file. 
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1.0959  mps  8  11/30/2012  12:11:20  AM 
1.3764  mps  @  11/30/2012  12:11:21  AM 
0.9190  mps  @  11/30/2012  12:11:22  AM 
0.7197  mps  8  11/30/2012  12:11:23  AM 
WIFI_CONN  8  11/30/2012  12:11:23  AM 
TX_START  8  11/30/2012  12:11:23  AM 
1.9180  mps  8  11/30/2012  12:11:24  AM 
0.5781  mps  8  11/30/2012  12:11:25  AM 
0.3186  mps  8  11/30/2012  12:11:26  AM 
0.7450  mps  8  11/30/2012  12:11:27  AM 
0.1642  mps  8  11/30/2012  12:11:28  AM 
0.7080  mps  8  11/30/2012  12:11:29  AM 
1.7338  mps  8  11/30/2012  12:11:30  AM 
WI FI_DI SCONNECT  8  11/30/2012  12:11:30  AM 
1.3015  mps  8  11/30/2012  12:11:31  AM 
1.5235  mps  8  11/30/2012  12:11:32  AM 
WI FI_C0NN  8  11/30/2012  12:11:32  AM 
0.8866  mps  8  11/30/2012  12:11:33  AM 
1.4841  mps  8  11/30/2012  12:11:34  AM 
TX_D0NE  8  11/30/2012  12:11:34  AM 
3.0644  mps  8  11/30/2012  12:11:35  AM 
4.0769  mps  8  11/30/2012  12:11:35  AM 
3.2224  mps  8  11/30/2012  12:11:36  AM 
3.5195  mps  8  11/30/2012  12:11:36  AM 
2.0872  mps  8  11/30/2012  12:11:37  AM 


Figure  18.  Sample  Log  File  Containing  Erroneous  Events 


Failed  assertions: 

JUnit  Test  JUnrtFromLogs.TestJogOutput 

Propositional  assertion  failures: 

None 

Statechart  assertion  failures: 

class  assertionrepository.lessThan2mps 

class  assertionrepository.logTxTimer 


Figure  19.  Failures  After  Using  the  Log  File 

Conclusion 

This  paper  presented  a  method  for  performing  V&V  on  a  mobile  application  using 
statechart  assertion  and  log  file-based  runtime  verification.  The  environment  that  the  DoD 
frequently  operates  in  is  abnormal,  to  say  the  least,  and  is  tough  to  emulate  when 
attempting  to  perform  V&V  in  a  lab  environment.  It  is  important  that  an  application  is 
evaluated  in  the  environment  in  which  it  is  expected  to  operate,  especially  since  the 
programmers  are  probably  unfamiliar  with  that  environment.  Log  files  provide  direct  insight 
into  the  operation  of  the  application,  and  when  used  in  the  expected  environment,  can 
ensure  a  thorough  and  valid  set  of  V&V  tests.  Combining  the  use  of  application  log  files  and 
statechart  assertions  allows  testers  to  evaluate  the  behavior  of  an  application  as  it  pertains 
to  its  adherence  to  the  stated  requirements.  Statechart  assertions  provide  a  mechanism  to 
represent  application  requirements  in  an  easy-to-follow  diagram  that  will  be  used  by 
StateRover  to  automatically  produce  executable  evaluators  to  evaluate  the  application  log 
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files.  The  modeling  of  the  requirements  independent  of  the  implementation  allows  for 
multiple  applications  to  be  evaluated  against  the  same  set  of  requirements. 

We  demonstrated  the  method  with  a  case  study  involving  the  V&V  of  a  GPS  mobile 
app.  There  are  two  different  services  one  can  use  to  get  the  user’s  current  location:  the 
standard  location  service  and  the  significant-change  location  service.  The  standard  location 
service  is  a  configurable,  general-purpose  solution  and  is  supported  in  all  versions  of  iOS. 
The  significant-change  location  service  offers  a  low-power  location  service  that  is  available 
only  in  iOS  4.0  and  later,  and  that  can  also  wake  up  an  app  that  is  suspended.  Initially  in  our 
case  study,  we  attempted  to  use  the  significant-change  location  service  to  generate  the  log 
file,  but  this  resulted  in  failure  of  the  statechart  assertions  for  the  speed-based  GPS  update 
requirements.  After  switching  to  the  standard  location  service  with  highest  accuracy  to 
generate  the  GPS  updates,  we  were  able  to  produce  a  new  log  file  that  satisfies  the 
statechart  assertions.  Note  that  it  would  be  very  labor  intensive  and  difficult  to  manually 
determine  if  the  new  log  file  meets  the  requirements  any  better  than  the  previous  version. 
The  StateRover’s  log  file-to-JUnit  converter  and  the  namespace  mapping  tool  significantly 
ease  the  task  of  the  checking  of  test  results;  we  can  quickly  see  that  the  new  log  file  (and 
hence  the  new  implementation)  does  indeed  meet  the  requirements,  once  we  have  imported 
the  log  file  into  StateRover.  The  methods  for  testing  mobile  apps,  as  discussed  in  The  V&V 
of  Mobile  Apps  in  this  paper,  all  require  the  manual  evaluation  of  test  results.  The  method 
put  forth  in  this  paper  not  only  automates  the  checking  of  test  results,  it  also  allows  testing  of 
the  application  in  the  expected  environment  of  operation.  The  case  study  provides  a  non¬ 
trivial  example  of  how  the  use  of  log  files  and  statechart  assertions  provide  a  significant 
improvement  in  the  V&V  process  of  applications. 
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